Deep Learning and Neural Networks

The phrase “neural network” can be intimidating, and the biological vocabulary doesn’t help: it suggests a mysterious, vaguely brain-like machine, out of reach without a PhD in mathematics. The reality is more modest and far more interesting. A neural network is a stack of multiplications, additions, and a simple function applied to the result. Nothing more.

What deserves to be understood is therefore not the formula for a neuron — it fits on one line — but two less obvious things. Why do we need to stack several layers when a single one would suffice if the world were simple? And how can a machine find millions of parameters on its own, with nobody telling it which direction to go?

The answer to the second question is called backpropagation, and it is regularly presented either as self-evident or as a wall of equations. We will take a third route: writing it out entirely in about thirty lines of NumPy, on a problem we can verify at a glance, and watching the error shrink step by step.

By the end of this lesson, you will have built a network that learns, without any deep learning library, and you will know exactly what each line does.

A Neuron: A Weighted Sum and a Decision

An artificial neuron takes several numbers as input, multiplies each by a weight, adds everything up, adds a bias, then passes the result through a function. That’s all.

import numpy as np

def neurone(entrees, poids, biais):
    somme = float(np.dot(entrees, poids) + biais)
    return 1 if somme > 0 else 0     # activation: simple threshold

# A neuron that computes logical AND.
poids = np.array([1.0, 1.0])
biais = -1.5

for x in ([0, 0], [0, 1], [1, 0], [1, 1]):
    print(x, neurone(np.array(x, dtype=float), poids, biais))
# [0, 0] 0
# [0, 1] 0
# [1, 0] 0
# [1, 1] 1

The arithmetic can be checked in your head: with both inputs at 1, the sum is 1 + 1 - 1.5 = 0.5, which is positive, so the output is 1. With only one input at 1, the sum is 1 - 1.5 = -0.5, which is negative, so the output is 0. The bias plays exactly the role of a firing threshold: you have to exceed 1.5 to activate the neuron.

This device is the perceptron, proposed by Frank Rosenblatt in 1957. Here the weights were set by hand; learning will consist of having the machine find them.

Why Stack Layers

A single neuron draws a straight boundary in the input space. As long as the classes can be separated by a line, that’s enough. The problem is that a case as elementary as the exclusive OR (XOR) cannot be.

      x2
       │
     1 │  ●(0,1)          ○(1,1)         ● = class 1
       │                                 ○ = class 0
       │
     0 │  ○(0,0)          ●(1,0)
       └──────────────────────────── x1
          0                1

   No line can put the two ● on one side
   and the two ○ on the other.

This is not just an intuition; it can be measured:

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([0, 1, 1, 0])            # exclusive OR

lineaire = LogisticRegression().fit(X, y)
print(lineaire.score(X, y), lineaire.predict(X))   # 0.5 [0 0 0 0]

reseau = MLPClassifier(hidden_layer_sizes=(4,), activation="tanh",
                       max_iter=5000, random_state=1).fit(X, y)
print(reseau.score(X, y), reseau.predict(X))       # 1.0 [0 1 1 0]

The linear model scores 0.5 — the score of chance on a balanced problem — and predicts the same class for all four points: it could do nothing. The network with one hidden layer of four neurons scores 1.0.

The mechanism is as follows: the hidden layer transforms the input space. In the new space it produces, the four points have become separable by a line, and the output layer only has to draw that line. Stacking layers means chaining changes of representation until the problem becomes simple.

This network has only 17 parameters: 2 × 4 input weights, 4 hidden biases, 4 × 1 output weights, 1 output bias. A modern network applies the same principle with several orders of magnitude more.

Activation Functions

The perceptron’s threshold has a fatal flaw for learning: its slope is zero everywhere except at one point where it is infinite. It’s impossible to extract a correction direction from it. So we replace it with smooth functions.

import numpy as np

z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])

print(np.maximum(0, z))                 # [0.  0.  0.  0.5 2. ]      ReLU
print((1 / (1 + np.exp(-z))).round(3))  # [0.119 0.378 0.5 0.622 0.881]  sigmoid
print(np.tanh(z).round(3))              # [-0.964 -0.462 0. 0.462 0.964]  tanh

The role of these functions is often misunderstood. They are not there to “normalize”: they are there to introduce nonlinearity. Without them, stacking ten layers of multiplications and additions would produce… one multiplication and one addition. A composition of linear functions is linear, so a network without activation, whatever its depth, would remain as powerless on XOR as the logistic regression above.

Gradient Descent: Correcting in the Right Direction

The central question remains: how do we find the weights? The idea is to measure the error, compute in which direction each parameter must move to reduce it, and take a small step in that direction. Then repeat.

Let’s revisit the rents from the previous lesson, but without LinearRegression: this time we write the learning ourselves.

import numpy as np

x = np.array([30.0, 45.0, 60.0, 75.0, 90.0])
y = np.array([620.0, 780.0, 1010.0, 1150.0, 1400.0])

# Scale the inputs: essential, we'll see why right after.
xn = (x - x.mean()) / x.std()

a, b = 0.0, 0.0
pas = 0.1

for etape in range(1, 201):
    prediction = a * xn + b
    erreur = prediction - y
    perte = (erreur ** 2).mean()          # mean squared error

    # Derivatives of the loss with respect to a and b.
    a -= pas * 2 * (erreur * xn).mean()
    b -= pas * 2 * erreur.mean()

    if etape in (1, 10, 50, 200):
        print(etape, round(perte, 1), round(a, 2), round(b, 2))

# 1   1059080.0  54.59  198.4
# 10  19587.4    243.64 885.48
# 50  518.0      272.94 991.99
# 200 518.0      272.94 992.0

The loss goes from over a million to 518 and stays there. Two checks show that the result is not approximate:

• The bias converges to 992.0, which is exactly the mean of the five rents: (620 + 780 + 1010 + 1150 + 1400) / 5 = 992.
• The slope converges to 272.94 in the scaled space. Converting back to square meters means dividing by the standard deviation of the areas, which is 21.213: 272.94 / 21.213 ≈ 12.86 € per m². This is the coefficient that LinearRegression found in the previous lesson, obtained here through guided trial and error.

The scaling is not cosmetic. With the raw areas and the same step size of 0.1, the descent diverges: the loss goes from 1.06 × 10⁶ to 6.9 × 10¹¹ by the second iteration, then explodes. The gradient is proportional to the magnitude of the inputs; inputs around 60 produce gigantic steps that overshoot the minimum instead of approaching it. This is the leading cause of a training run that “doesn’t converge.”

loss
  │╲
  │ ╲        step too small: correct but slow
  │  ╲___
  │      ╲___
  │          ╲________
  └────────────────────► iterations

loss
  │   ╱╲    ╱╲
  │  ╱  ╲  ╱  ╲   step too large: oscillates or explodes
  │ ╱    ╲╱    ╲
  └────────────────────► iterations

Backpropagation, Without the Formalism

On a model with two parameters, we write the derivatives by hand. In a network, a weight in the first layer influences the output through all the following layers. Backpropagation is the way of organizing this computation: we propagate the error from the output back to the input, layer by layer, reusing at each step what has already been computed.

Here is a complete network — two inputs, four hidden neurons, one output — written without any deep learning library.

import numpy as np

rng = np.random.default_rng(1)

X = np.array([[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]])
y = np.array([[0.0], [1.0], [1.0], [0.0]])       # exclusive OR

# Random initialization: two identical weights would always receive
# the same correction and would remain identical forever.
W1 = rng.normal(0, 1, (2, 4))
b1 = np.zeros((1, 4))
W2 = rng.normal(0, 1, (4, 1))
b2 = np.zeros((1, 1))

def sigmoide(z):
    return 1 / (1 + np.exp(-z))

for etape in range(1, 5001):
    # --- Forward pass: from the input to the prediction ---
    z1 = X @ W1 + b1
    a1 = np.tanh(z1)
    z2 = a1 @ W2 + b2
    a2 = sigmoide(z2)

    perte = float(np.mean((a2 - y) ** 2))

    # --- Backward pass: from the error to the weights ---
    # d2: sensitivity of the loss to z2. The factor a2*(1-a2) is the
    # derivative of the sigmoid; it expresses "how much the output moves
    # if z2 moves a little".
    d2 = 2 * (a2 - y) / len(X) * a2 * (1 - a2)
    dW2 = a1.T @ d2
    db2 = d2.sum(axis=0, keepdims=True)

    # We go back one layer: the error is redistributed to the hidden
    # neurons in proportion to their weight W2, then multiplied by
    # the derivative of tanh, which equals 1 - tanh².
    d1 = (d2 @ W2.T) * (1 - a1 ** 2)
    dW1 = X.T @ d1
    db1 = d1.sum(axis=0, keepdims=True)

    # --- Update ---
    for param, gradient in ((W1, dW1), (b1, db1), (W2, dW2), (b2, db2)):
        param -= 0.5 * gradient

    if etape in (1, 100, 1000, 5000):
        print(etape, round(perte, 4))

# 1     0.24
# 100   0.1182
# 1000  0.0018
# 5000  0.0003

print(a2.round(3).ravel())    # [0.007 0.983 0.983 0.021]

The network starts with an error of 0.24 and drops to 0.0003. The four final outputs — 0.007 / 0.983 / 0.983 / 0.021 — are to be compared with the targets 0 / 1 / 1 / 0. XOR is solved by a program whose entire learning logic fits in eight lines.

Three observations are worth remembering:

• The backward pass mirrors the forward pass. X @ W1 on the way forward becomes X.T @ d1 on the way back. This is no coincidence: it is the chain rule applied to a matrix product.
• No layer knows what it detects. The hidden layer built a useful representation of the four points, but nobody told it which one. This is what is called representation learning, and it is the main contribution of deep learning.
• Nothing here depends on the number of layers. With ten layers, you repeat the same block ten times. It is this uniformity that made deep networks possible, and libraries like PyTorch or TensorFlow do nothing more than automatically derive these backward passes.

What Made Deep Learning Possible

Backpropagation was popularized in 1986 by Rumelhart, Hinton, and Williams. The rise of deep learning dates from the 2010s. This twenty-five-year gap is not due to a missing theoretical discovery, but to three material conditions that were not yet met.

The multiplier is brutal. Each iteration of our loop processes 4 examples and updates 17 parameters. Training a typical vision network processes millions of images and updates tens of millions of parameters, tens of times over. The theory is the same; it is the scale that changes in nature, and with it the cost, whose consequences we will measure in the final lesson.

Key Takeaways

• A neuron is a weighted sum followed by a function. The bias acts as a threshold, and the arithmetic can be checked by hand on logical AND.
• A single layer draws only a straight boundary: it fails on XOR with a score of 0.5. Stacking layers serves to transform the space until the problem becomes linearly separable.
• Activation functions exist to introduce nonlinearity. Without them, a ten-layer network is equivalent to a single layer, regardless of its size.
• Gradient descent measures the error, computes the correction direction, and takes a small step. The step size and the scale of the inputs are critical: the same unscaled data makes the same loop diverge.
• Backpropagation is just the chain rule organized from the output back to the input. It fits in a few lines, it is identical regardless of the number of layers, and that is what makes it industrializable.