Computer Vision

Recognizing a cat in a photo is, for a human, instantaneous and unconscious. It is also the problem that resisted computing the longest. For decades, researchers hand-wrote detectors for edges, corners, and textures, and stacked up rules to combine them, without ever approaching the robustness of a three-year-old child.

The difficulty is not a lack of information: it is an excess of it. A phone photograph contains several million numbers, none of which means anything in isolation. The value of a pixel says nothing; it is the local organization of thousands of pixels that forms an edge, and the organization of thousands of edges that forms a face. And this organization must be recognized regardless of the object’s position, lighting, scale, or orientation.

The idea that unlocked the problem is surprisingly economical. Rather than learning a rule for every possible position in the image, you learn one small detector and slide it everywhere. A pattern learned once is recognized wherever it appears, and the number of parameters to tune collapses. This operation is called convolution, and we are going to write it ourselves in about ten lines.

We will then see how these detectors are stacked, how the three major vision tasks really differ, and why almost nobody trains a vision model from scratch today.

An image is an array of numbers

A grayscale image is a matrix: one intensity value per pixel. A color image is three stacked matrices, one for each red, green, and blue channel.

from sklearn.datasets import load_digits

chiffres = load_digits()
print(chiffres.images.shape)   # (1797, 8, 8)
print(chiffres.images.min(), chiffres.images.max())   # 0.0 16.0
print(chiffres.target[0])      # 0

print(chiffres.images[0].astype(int))
# [[ 0  0  5 13  9  1  0  0]
#  [ 0  0 13 15 10 15  5  0]
#  [ 0  3 15  2  0 11  8  0]
#  [ 0  4 12  0  0  8  8  0]
#  [ 0  5  8  0  0  9  8  0]
#  [ 0  4 11  0  1 12  7  0]
#  [ 0  2 14  5 10 12  0  0]
#  [ 0  0  6 13 10  0  0  0]]

If you squint, you can make out the zero: two columns of high values separated by a central hollow of zeros. This is exactly what the machine receives, nothing more and nothing less.

Scale is the first obstacle. This dataset uses 8 × 8 thumbnails, or 64 numbers per image. A typical input photograph for a vision network is 224 × 224 pixels with 3 channels:

print(224 * 224 * 3)   # 150528

150,528 numbers for a single image. Everything that follows is about handling this volume without blowing up the number of parameters to learn.

Convolution: detecting a pattern, wherever it is

A convolution slides a small matrix of weights, the kernel (or filter), across the image. At each position, it multiplies element by element and sums the results. The output is a new image, called an activation map, that shows where the sought pattern is present.

import numpy as np

# Une barre verticale claire sur fond sombre.
image = np.array([
    [0, 0, 9, 9, 0, 0],
    [0, 0, 9, 9, 0, 0],
    [0, 0, 9, 9, 0, 0],
    [0, 0, 9, 9, 0, 0],
], dtype=float)

# Noyau de Sobel simplifié : détecteur de contour vertical.
noyau = np.array([[1.0, 0.0, -1.0],
                  [1.0, 0.0, -1.0],
                  [1.0, 0.0, -1.0]])

def convoluer(image, noyau):
    h, w = image.shape
    kh, kw = noyau.shape
    sortie = np.zeros((h - kh + 1, w - kw + 1))
    for y in range(sortie.shape[0]):
        for x in range(sortie.shape[1]):
            fenetre = image[y:y + kh, x:x + kw]
            sortie[y, x] = float((fenetre * noyau).sum())
    return sortie

carte = convoluer(image, noyau)
print(carte)
# [[-27. -27.  27.  27.]
#  [-27. -27.  27.  27.]]

Let’s verify one value by hand. At the top right, the window covers columns 3, 4, 5, that is the values [[9,0,0],[9,0,0],[9,0,0]]. The product with the kernel gives 9×1 + 0×0 + 0×(−1) repeated over three rows, or 27. The value +27 signals a light → dark transition. On the left, the same transition in the opposite direction gives −27. At the center of the pattern, a symmetric kernel would give 0: nothing to report.

Three properties give the operation all its power:

• Locality. Each output depends only on a small neighborhood, which matches the real structure of images: an edge is a local phenomenon.
• Weight sharing. The nine kernel weights are the same at every position. A detector learned once works everywhere in the image.
• Economy. This is the quantified consequence of the previous point, and it is spectacular.

# Une couche dense reliant tous les pixels d'une image 224×224×3 à 1000 neurones :
print(150528 * 1000 + 1000)      # 150529000

# Une couche convolutive de 64 noyaux de 3×3 sur 3 canaux :
print(3 * 3 * 3 * 64 + 64)       # 1792

150 million parameters versus 1,792. A factor of roughly 84,000. Without weight sharing, deep learning on images would never have been practical, and it is not a matter of patience: the number of examples needed to fit 150 million parameters in a single layer is out of reach.

Shrinking, stacking, and seeing wider

A single convolution only looks at a 3 × 3 neighborhood. To recognize an entire object, the field of view has to widen progressively. Two mechanisms do this.

The first is pooling, whose most common form keeps the maximum of each block:

def pooling_max(carte, taille=2):
    h, w = carte.shape
    return carte[:h - h % taille, :w - w % taille].reshape(
        h // taille, taille, w // taille, taille).max(axis=(1, 3))

print(pooling_max(np.abs(carte)))   # [[27. 27.]]

The 2 × 4 map becomes 1 × 2. We lose the exact position of the edge and keep its intensity: this is precisely the trade-off we want for classification, where knowing that there is an edge matters more than its pixel-exact position.

The second mechanism is stacking. Each layer sees the outputs of the previous one, and therefore an increasingly wide neighborhood of the original image:

Couche 1  (3×3 pixels vus)    →  contours, taches de couleur
Couche 2  (~7×7 pixels vus)   →  coins, angles, courbes
Couche 3  (~15×15)            →  motifs : œil, roue, texture
Couche 4+ (une grande partie) →  parties d'objets, puis objets

This hierarchy is not imposed by the programmer: it emerges from training, and it has been found experimentally in trained networks by visualizing what activates each filter the most. This is the representation learning seen in the previous lesson, applied to images.

Two dates structure this history: LeNet-5, published by Yann LeCun et al. in 1998, already applied this scheme to reading checks; AlexNet, in 2012, scaled it up to ImageNet and set off the current wave.

Classifying, detecting, segmenting

These three words refer to tasks that are often confused. They are distinguished unambiguously by the shape of their output, which determines the cost of annotation and the difficulty of the problem.

print(224 * 224 * 20)   # 1003520

The gap between 20 and a million output values explains most of it: segmentation requires pixel-level annotations, whose production cost is incomparable to a simple image label.

In detection, the reference metric measures the overlap between the predicted box and the ground-truth box. It can be computed exactly:

def iou(a, b):
    """Intersection sur union de deux boîtes (x1, y1, x2, y2)."""
    x1 = max(a[0], b[0]); y1 = max(a[1], b[1])
    x2 = min(a[2], b[2]); y2 = min(a[3], b[3])
    inter = max(0, x2 - x1) * max(0, y2 - y1)
    aire_a = (a[2] - a[0]) * (a[3] - a[1])
    aire_b = (b[2] - b[0]) * (b[3] - b[1])
    return inter / (aire_a + aire_b - inter)

print(round(iou((10, 10, 50, 50), (30, 30, 70, 70)), 3))   # 0.143
print(round(iou((10, 10, 50, 50), (12, 12, 52, 52)), 3))   # 0.822

First case: two 40 × 40 boxes offset by 20 pixels overlap over 20 × 20 = 400 pixels, for a union of 1600 + 1600 − 400 = 2800, or 400 / 2800 ≈ 0.143. The usual convention considers a detection correct above 0.5. The second box, offset by only 2 pixels, overlaps over 38 × 38 = 1,444 pixels for a union of 1,756, or 0.822: correct.

Transfer learning

Training a vision network from scratch requires a lot of annotated images (ImageNet provides about 1.2 million for 1,000 categories) and a substantial compute budget. Almost nobody has that for a specific business problem: classifying defective parts, sorting X-rays, recognizing a plant species.

Transfer learning solves this problem in two steps. You take a network already trained on a large corpus, freeze its layers (they have learned detectors for edges, textures, and patterns that remain relevant for other images), and retrain only the last layer, the one that makes the decision.

The savings can be quantified. With a backbone producing a vector of 2,048 features and 10 target classes, the layer to learn has 2048 × 10 + 10 = 20,490 parameters, instead of the tens of millions in the full network.

It remains to be seen whether the principle holds up. Let’s test it honestly: we “pre-train” a representation on digits 5 to 9, then learn to recognize digits 0 to 4 with only 5 examples per class.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)

source = X[y >= 5]                       # « pré-entraînement » : classes 5 à 9
cible_X, cible_y = X[y < 5], y[y < 5]    # tâche cible : classes 0 à 4

Xp, Xt, yp, yt = train_test_split(cible_X, cible_y, test_size=0.4,
                                  random_state=0, stratify=cible_y)

indices = np.concatenate([np.where(yp == c)[0][:5] for c in range(5)])

brut = LogisticRegression(max_iter=5000).fit(Xp[indices], yp[indices])
print(round(brut.score(Xt, yt), 3))                       # 0.936

# La représentation est apprise sur les classes 5-9 uniquement,
# sans jamais voir une étiquette de la tâche cible.
pca = PCA(n_components=30, random_state=0).fit(source)
tete = LogisticRegression(max_iter=5000).fit(pca.transform(Xp[indices]), yp[indices])
print(round(tete.score(pca.transform(Xt), yt), 3))         # 0.931

The result deserves a frank comment: the transferred representation does no better than the raw pixels (0.931 versus 0.936). The experiment does not demonstrate the benefit of transfer, and it would be dishonest to present it as if it did.

This negative result is instructive. Transfer brings nothing here for three identifiable reasons: an 8 × 8 thumbnail contains only 64 values, already directly usable; principal component analysis is a linear transformation, incapable of building the hierarchy of patterns that gives a convolutional backbone its value; and 30 components out of 64 dimensions compress almost nothing. Transfer produces its documented gains when the backbone is deep, nonlinear, and pre-trained on images far richer than these thumbnails. The mechanics shown above, freezing the representation and learning only the head, are what is used in practice; the scale, however, is not what makes it pay off.

Key takeaways

• An image is an array of numbers, and the problem is how many there are: 150,528 values for a single 224 × 224 color photograph.
• Convolution slides a small kernel of shared weights across the whole image. This sharing is what makes it all feasible: 1,792 parameters instead of 150 million for an equivalent dense layer.
• Pooling and stacking layers progressively widen the field of view and let a hierarchy emerge (edges, patterns, object parts) that nobody specified.
• Classification, detection, and segmentation differ by the shape of their output, from 20 numbers to more than a million. The cost of annotation follows the same progression.
• Transfer learning consists of freezing a pre-trained backbone and retraining only the decision layer, about 20,000 parameters instead of millions. Its benefit depends on the richness of the backbone: on 8 × 8 thumbnails with a linear transformation, the experiment above shows none.