The math, intuition, and code behind how neural networks actually learn — built up from a single neuron to a working training loop. Based on Andrej Karpathy's micrograd tutorial.
Companion to Part 1: How LLMs Work. All concepts and code traced directly to Karpathy's micrograd lecture.
Before we build anything, let's understand what we're trying to do. We have some inputs and we want to predict an output. For example: given these 4 measurements about a person, predict whether they'll like a movie.
The challenge: we don't know the formula. We can't write the rules by hand. Instead, we want a system that learns the formula from examples — just by seeing a lot of input/output pairs.
A neural network is that system. It starts as a completely random function. Then it looks at thousands of examples and adjusts itself — tiny nudge by tiny nudge — until its predictions get good. This process is called training.
A tiny example to make it concrete. Four inputs, one target output:
xs is our input data — 4 examples, each with 4 numbers. ys is the "right answer" for each example: 1.0 or -1.0. The network starts knowing nothing, and has to learn to map xs to ys.
A neuron is just a tiny mathematical function. Think of it as a dimmer switch: it takes a bunch of inputs, decides how much each one matters (the weights), adds a personal default lean (the bias), and squishes the result to a bounded range.
The formula: output = tanh(w₁·x₁ + w₂·x₂ + b)
Where w₁, w₂ are weights ("how much does this input matter?"), b is the bias ("what's my default lean when inputs are zero?"), and tanh squishes the result to always land between -1 and 1.
self.data is just a number — like 0.5 or -1.3. self.grad starts at zero and gets filled in during backpropagation (§7). The __mul__ and __add__ methods let us use normal Python math operators (+, *) with Value objects. tanh squishes any number to the range (-1, 1).
Drag the sliders to change the weights and bias. Watch the neuron's output update live. Fixed inputs: x₁ = 0.5, x₂ = −0.3.
One neuron isn't enough to learn complex patterns. We stack them into layers, and stack layers into a Multi-Layer Perceptron (MLP). Think of it as an assembly line: the first layer looks at raw inputs, the next layer looks at what the first layer found, and so on.
Every connection between neurons is one weight. A network with 4 inputs → 3 neurons → 1 output has (4×3) + (3×1) = 15 weights, plus biases. GPT-4 has the same structure, just with 405 billion weights.
Neuron(nin) creates a neuron with nin random weights. Layer(nin, nout) creates nout neurons that each accept nin inputs. MLP(4, [3,1]) creates a 4→3→1 network. __call__ is Python's way of making an object callable — so model(x) runs the forward pass.
A 4→3→1 network: 4 inputs, one hidden layer of 3 neurons, 1 output.
The forward pass is simply: run data through the network from left to right. Each neuron fires in turn, passes its output to the next layer, and eventually we get a prediction out the end.
At the start, since weights are random, predictions are meaningless. But we can still run the forward pass — we need it to compute how wrong we are (§5), which tells us how to improve.
n(x) calls the MLP's __call__ method, which loops through each layer and applies it to x. Each layer call applies each neuron: compute w·x + b, apply tanh, pass output to next layer. The final value is our prediction. It's just a chain of multiplications and additions.
Click "Run Forward Pass" to watch data flow through a 2→3→1 network. Each node lights up with its computed value.
After the forward pass, we have predictions. Now we need to measure how wrong they are. The loss function is a report card: one single number that summarizes "here is how bad your predictions are right now."
We use Mean Squared Error (MSE): for each example, square the difference between our prediction and the target, then average them all. Squaring ensures the loss is always positive and punishes big mistakes harder.
The goal of training: make this loss number as small as possible.
yout is our network's prediction, ygt is the "ground truth" target. (yout - ygt)**2 squares the error. If we predicted 0.23 but the target was 1.0, the error is (0.23−1.0)² = 0.59. We sum all 4 example errors to get one number. The training job is to push this number toward zero.
The loss is a function of all the weights. Think of it as a hilly terrain — we want to roll the ball to the lowest point. Click "Step" to run one gradient descent update.
We want to push the loss down. To do that, we need to know: for each weight, does increasing it make the loss go up or down? That's exactly what a derivative tells us.
Imagine standing on a hill. You can't see the whole landscape, but you can feel which direction your foot is going downhill. The derivative is that "feel" — the slope of the loss at your current position.
Key rules we'll use:
Drag the point along f(x) = x². The tangent line shows the slope (derivative) at that position. Notice: at x=0 the slope is 0; it gets steeper as x moves away from center.
We know the loss. We know derivatives. But we have hundreds of weights — how do we know which ones to blame, and by how much?
Backpropagation solves this efficiently. It walks the computation graph backwards — from the loss, through every operation, back to every weight — computing each weight's gradient using the chain rule.
Karpathy's key insight: "The only thing backprop does is apply the chain rule recursively." Nothing more. It's mechanical, not magical.
p.grad = 0.0 clears leftover gradients from the previous training step — they'd otherwise accumulate. loss.backward() walks backwards through every operation that produced loss, applying the chain rule at each node. After this call, every weight has a .grad that says "increase me → loss increases by this much."
A simple expression: e = tanh((a·b) + a). Click "Forward Pass" to compute values, then "Backprop" to watch gradients flow backwards.
We have gradients. Now we use them. The update rule is simple: for each weight, move it a tiny step in the opposite direction of its gradient. Opposite because the gradient points uphill — we want to go downhill.
.grad via chain rule. Update — p.data -= 0.01 * p.grad moves each weight a tiny step opposite its gradient. 0.01 is the learning rate. Repeat 100 times: loss drops from ~4 to near 0.
Built from Andrej Karpathy's "The spelled-out intro to neural networks and backpropagation: building micrograd" lecture. Now you understand the math behind Part 1's training section at the code level.
← Part 1: How LLMs Work · ← Part 2: How to Use LLMs · Source lecture