Machine Learning: The Basics

A model that gets 99% of its answers right can be completely useless. A model that perfectly recognizes every example it was built on can be the worst of the lot. Both statements sound absurd, yet they are everyday reality in machine learning, and ignoring them is the most common way to ship a system that fails the moment it goes live.

The reason is that machine learning is not trying to memorize examples: it is trying to generalize to cases it has never seen. But you cannot measure that ability directly — by definition, future cases are not available. The whole art, then, lies in setting up an honest indirect measurement, and then choosing the right indicator to read it.

This lesson builds these safeguards in the order you need them. We’ll first look at what “learning” concretely means for a machine, which comes down to adjusting numbers. Then the three major families of problems, depending on what information you have. Next, the protocol that lets you measure generalization, and the phenomenon it reveals: overfitting. Finally, the choice of metrics, where most misunderstandings between technical and business teams occur.

All examples use scikit-learn and datasets bundled with the library: they run as-is, with no download required.

Learning Means Adjusting Parameters

A model is a parameterized function. Learning means searching for the parameter values that minimize the gap between predictions and known answers. It is no more mysterious than that.

import numpy as np
from sklearn.linear_model import LinearRegression

# Five apartments: area in m², rent in euros.
surface = np.array([[30], [45], [60], [75], [90]])
loyer = np.array([620, 780, 1010, 1150, 1400])

modele = LinearRegression().fit(surface, loyer)

print(round(float(modele.coef_[0]), 2))      # 12.87
print(round(float(modele.intercept_), 2))    # 220.0
print(modele.predict([[50]]).round(1))       # [863.3]

The model found two numbers. The learned rule reads in plain English: about €12.87 per square meter, plus a €220 fixed fee. It wasn’t written by hand; it was inferred from five observations by minimizing the sum of squared errors.

What matters here is the nature of the prediction for 50 m²: €863.30. No 50 m² apartment appears in the data. The model interpolated between the 45 m² and the 60 m² apartments. That’s legitimate. By contrast, asking it for the rent of a 400 m² apartment would be an extrapolation far beyond the observed range: the formula would answer without hesitation, and there would be no reason to believe it.

Four terms to pin down, because they come up everywhere:

The Three Families of Learning

The family depends on the information you have, not on the algorithm.

Supervised Learning

You have labeled examples: for each input, the correct answer. That’s the case for the regression above, and for any classification. It is by far the most widely used setting in production, because it is the easiest to measure — and the most expensive, because you have to produce the labels.

Unsupervised Learning

You have only the inputs. The algorithm looks for structure: groups, axes of variation, anomalies. Evaluation becomes tricky, since there is no ground truth.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris

iris = load_iris()
# We do NOT use iris.target: the algorithm sees no labels.
groupes = KMeans(n_clusters=3, n_init=10, random_state=0).fit_predict(iris.data)

print(np.bincount(groupes))   # [62 50 38]

# After-the-fact comparison with the true species, to judge the result.
import collections
print(collections.Counter(zip(groupes.tolist(), iris.target.tolist())))

The final tally shows exactly what you need to understand about unsupervised learning. One cluster contains all 50 setosa and nothing else: that species is clearly separated from the other two. The two remaining clusters mix versicolor and virginica: 14 virginica end up with the versicolor, and 2 versicolor with the virginica. Sixteen flowers out of one hundred fifty are grouped incorrectly, a little over 10%.

The algorithm didn’t fail: it found the three most compact groups in the Euclidean-distance sense. Those groups don’t perfectly coincide with the species because nothing forced them to. That is the intrinsic limit of unsupervised learning: it reveals a structure, not the structure you had in mind.

Reinforcement Learning

Neither labeled inputs nor mere observation: an agent acts, receives a numerical reward, and must discover the strategy that maximizes the cumulative total of those rewards. The difficulty specific to this setting is the trade-off between exploiting what seems good and exploring what hasn’t been tried enough.

import numpy as np

# Three levers, each with a win probability unknown to the agent.
rng = np.random.default_rng(0)
vraies = np.array([0.2, 0.5, 0.75])

estimations = np.zeros(3)
tirages = np.zeros(3)

for t in range(2000):
    if rng.random() < 0.1:                    # 10% of the time: explore
        bras = rng.integers(3)
    else:                                     # otherwise: exploit the best known
        bras = int(np.argmax(estimations))

    recompense = 1.0 if rng.random() < vraies[bras] else 0.0
    tirages[bras] += 1
    # Mean updated incrementally.
    estimations[bras] += (recompense - estimations[bras]) / tirages[bras]

print(estimations.round(2))     # [0.19 0.46 0.75]
print(tirages.astype(int))      # [  81   72 1847]

The final estimates approach the true probabilities, and above all the distribution of pulls tells the story of the learning: 1,847 trials out of 2,000 were devoted to the best lever, versus about 75 for each of the other two. The agent didn’t just estimate; it concentrated its budget. The price paid is called regret: the 153 trials on the mediocre levers were necessary to be sure they were mediocre.

Training and Testing: Why We Hide Data

Evaluating a model on the examples used to fit it measures its memory, not its ability to generalize. The rule, therefore, is to set aside part of the data before any learning, and to look at it only at the end.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

donnees = load_breast_cancer()      # 569 tumors, 30 measurements, 2 classes
X_train, X_test, y_train, y_test = train_test_split(
    donnees.data, donnees.target, test_size=0.3, random_state=42
)
print(X_train.shape, X_test.shape)  # (398, 30) (171, 30)

for profondeur in (1, 3, None):
    arbre = DecisionTreeClassifier(max_depth=profondeur, random_state=0)
    arbre.fit(X_train, y_train)
    print(profondeur,
          round(arbre.score(X_train, y_train), 3),
          round(arbre.score(X_test, y_test), 3))

The random_state parameter fixes the random draw: without it, two runs would give two different splits and figures that can’t be compared.

Overfitting, Measured

The table produced by the previous loop is the most instructive result of this lesson:

The unlimited-depth tree classifies all 398 training examples perfectly. Yet it performs worse on the test set than the depth-3 tree. It built branches that describe quirks of those specific 398 patients — noise, not a regularity. This is overfitting.

At the other extreme, the depth-1 tree asks only a single question: it is too rigid to capture what is actually there. This is underfitting. The right setting lies in between, and you find it by measurement, not by intuition.

error
  │
  │╲                                    ╱  ← test error
  │ ╲                                 ╱
  │  ╲                             ╱
  │   ╲__________________________╱        ← useful zone
  │                ╲
  │                  ╲___________________ ← training error
  └──────────────────────────────────────► model complexity
     underfitting            overfitting

A 0.076 gap between training and test is the warning sign to watch for. But a single split is still a random draw: measuring on 171 examples leaves a noticeable margin of uncertainty. Cross-validation repeats the operation over several splits:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(DecisionTreeClassifier(max_depth=3, random_state=0),
                         donnees.data, donnees.target, cv=5)
print(scores.round(3))              # [0.912 0.895 0.939 0.947 0.894]
print(scores.mean().round(3))       # 0.917
print(scores.std().round(3))        # 0.022

The mean (0.917) is a more sober estimate than the 0.965 obtained on a favorable split, and the standard deviation (0.022) quantifies the margin. Announcing “96.5%” based on a single split isn’t a lie, but it’s too fragile a measurement to base a decision on.

Choosing the Metric

Here is the trap announced at the start. For a disease affecting 1% of the population, a model that answers “healthy” for everyone achieves 99% accuracy.

import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score

# 1,000 patients, 10 of them sick.
y_vrai = np.array([0] * 990 + [1] * 10)

# Lazy model: "nobody is sick".
y_paresseux = np.zeros(1000, dtype=int)
print((y_vrai == y_paresseux).mean())                          # 0.99
print(recall_score(y_vrai, y_paresseux, zero_division=0))      # 0.0

99% accuracy, 0% recall: the model detects no sick patients at all. For a screening test, that’s a system with no value whatsoever. Accuracy is misleading as soon as classes are imbalanced, which is the case for almost every interesting problem — fraud, equipment failure, rare disease, manufacturing defects.

Let’s compare with an imperfect but useful model:

y_predit = np.zeros(1000, dtype=int)
y_predit[:20] = 1        # 20 healthy people wrongly flagged
y_predit[990:998] = 1    # 8 of the 10 sick patients correctly detected

print(confusion_matrix(y_vrai, y_predit))
# [[970  20]
#  [  2   8]]
print(round(precision_score(y_vrai, y_predit), 3))   # 0.286
print(round(recall_score(y_vrai, y_predit), 3))      # 0.8
print(round(f1_score(y_vrai, y_predit), 3))          # 0.421

The confusion matrix reads as follows: 970 healthy patients correctly classified, 20 false alarms, 2 sick patients missed, 8 sick patients detected.

This model has a lower accuracy than the lazy model (0.978 versus 0.990), and it is incomparably more useful: it finds 8 out of 10 sick patients. The price is 20 false alarms.

This trade-off has no universal solution. It is settled outside of mathematics:

• Screening for a serious but treatable disease: favor recall. A false alarm costs a follow-up test; a missed case can cost a life.
• Automatically blocking a bank account: favor precision. Every false positive is a legitimate customer who gets blocked.

That is why the metric must be chosen with the business side before training. Optimizing first and discussing afterward means redoing the work.

Key Takeaways

• Learning amounts to adjusting parameters to minimize a gap measured on examples. What isn’t in the data won’t be in the model.
• The three families are distinguished by the information available: labels (supervised), inputs only (unsupervised), rewards (reinforcement). Unsupervised learning reveals a structure, not necessarily the one you were looking for — 16 irises out of 150 grouped incorrectly serve as a reminder.
• A score on the training data measures nothing useful. The tree with 1.000 on training and 0.924 on test is worse than the one with 0.970 / 0.965: that’s overfitting.
• Cross-validation replaces a fragile score with a mean and a standard deviation. It’s the minimum before announcing a number to anyone.
• Accuracy is misleading on imbalanced classes: 99% while detecting no sick patients. Precision and recall are traded off according to the real cost of a false alarm versus that of a missed case, a decision that belongs to the business.