How Neural Networks Learn: Neurons, Layers and Backpropagation

Neural networks power image recognition, speech recognition, translation and large language models. The core idea is simple: a stack of small, adjustable calculations whose settings are tuned by gradient descent until the outputs are right.

A single neuron#

A neuron takes inputs, multiplies each by a weight, adds a bias, and passes the total through an activation function:

import numpy as np

def relu(z):
    return np.maximum(0, z)

inputs = np.array([2.0, -1.0, 0.5])
weights = np.array([0.4, 0.3, -0.6])
bias = 0.1
z = inputs @ weights + bias
print("weighted sum:", round(z, 2))
print("after ReLU:", round(relu(z), 2))

weighted sum: 0.3
after ReLU: 0.3

With a sigmoid activation and a single neuron, this is exactly logistic regression. The power comes from connecting many neurons.

Activation functions#

Without a non-linear activation, stacking layers would still produce a straight-line (linear) model. Common choices:

import numpy as np

z = np.array([2.0, 1.0, 0.1])
softmax = np.exp(z) / np.exp(z).sum()
print("softmax:", softmax.round(3), "sum:", softmax.sum().round(3))
print("sigmoid of 0 and 3:", (1 / (1 + np.exp(-np.array([0.0, 3.0])))).round(3))

softmax: [0.659 0.242 0.099] sum: 1.0
sigmoid of 0 and 3: [0.5   0.953]

Softmax turns any scores into probabilities that sum to 1, with the largest score getting the largest probability.

Layers#

Neurons are arranged in layers. Each neuron in a layer receives every output of the previous layer. A whole layer is one matrix multiplication:

inputs (4 features) → hidden layer (8 neurons, ReLU) → output layer (3 neurons, softmax)
weights: a 4×8 matrix and a 8×3 matrix, plus a bias per neuron

The forward pass runs inputs through the layers to get a prediction.

How it learns#

1. Forward pass: compute predictions.
2. Loss: measure how wrong they are. Classification uses cross-entropy (large when the model is confidently wrong); regression uses mean squared error.
3. Backpropagation: use the chain rule of calculus to compute the gradient of the loss with respect to every weight, meaning which direction each weight should move to reduce the loss.
4. Gradient descent: nudge every weight a small step (the learning rate) in that direction.
5. Repeat for many batches of data. One pass over the full dataset is an epoch.

A tiny network in NumPy#

XOR is the classic problem a straight line cannot solve: the output is 1 when exactly one input is 1. A network with one small hidden layer learns it:

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

rng = np.random.default_rng(1)
W1, b1 = rng.normal(0, 1, (2, 4)), np.zeros(4)
W2, b2 = rng.normal(0, 1, (4, 1)), np.zeros(1)
sigmoid = lambda z: 1 / (1 + np.exp(-z))
lr = 1.0

for epoch in range(1, 3001):
    h = np.tanh(X @ W1 + b1)                  # forward pass: hidden layer
    p = sigmoid(h @ W2 + b2)                  # forward pass: output probability
    loss = -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

    dz2 = (p - y) / len(X)                    # backpropagation (chain rule)
    dW2, db2 = h.T @ dz2, dz2.sum(axis=0)
    dz1 = (dz2 @ W2.T) * (1 - h ** 2)
    dW1, db1 = X.T @ dz1, dz1.sum(axis=0)

    W1 -= lr * dW1; b1 -= lr * db1            # gradient descent step
    W2 -= lr * dW2; b2 -= lr * db2
    if epoch in (1, 100, 1000, 3000):
        print(f"epoch {epoch:>4}: loss {loss:.4f}")

print("predictions:", p.ravel().round(3))

epoch    1: loss 0.6712
epoch  100: loss 0.0385
epoch 1000: loss 0.0024
epoch 3000: loss 0.0008
predictions: [0.    0.999 0.999 0.001]

The loss falls and the predictions approach 0, 1, 1, 0.

Starting weights matter. With default_rng(0) instead of 1, this same network gets stuck in a poor solution and predicts 0.5 for two of the four inputs. Training can settle in such a local minimum; larger networks, better initialisation and optimisers such as Adam make this much rarer in practice. Frameworks such as PyTorch compute the gradients automatically, so you never write the backpropagation lines by hand; you will see that in the next lessons.

A neural network in scikit-learn#

For small tabular problems, scikit-learn's MLPClassifier (multi-layer perceptron) is enough:

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

model = make_pipeline(StandardScaler(),
                      MLPClassifier(hidden_layer_sizes=(64,), max_iter=300, random_state=0))
model.fit(X_train, y_train)
print("test accuracy:", round(model.score(X_test, y_test), 3))
print("iterations run:", model[-1].n_iter_)

test accuracy: 0.973
iterations run: 172

Neural networks need scaled inputs: large unscaled values make training slow and unstable.

When to use neural networks#

• Images, audio, text and other unstructured data: neural networks are the clear choice.
• Tabular data (rows and columns): gradient-boosted trees usually match or beat them with less tuning.
• They need more data, more computing power and more tuning than classic models, and are harder to explain.

Common mistakes#

Practice#

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
for layers in [(16,), (32, 16)]:
    model = make_pipeline(StandardScaler(), MLPClassifier(hidden_layer_sizes=layers, max_iter=1000, random_state=0))
    print(layers, round(cross_val_score(model, X, y, cv=5).mean(), 3))

(16,) 0.975
(32, 16) 0.975

Key takeaway#

Next: PyTorch Tensors and Autograd.