Tests in the Continuous Integration Pipeline

Your CI pipeline takes twenty-six minutes. A developer opens a pull request, starts the run, and moves on to something else. Half an hour later, an end-to-end test is red; they rerun it without looking, because "that one fails every once in a while." Fifty minutes after the first commit, the merge goes through. With five pull requests a day, the team has lost four hours of waiting and, more importantly, the habit of reading failures.

This is the tipping point to avoid, and it has nothing to do with the quality of the individual tests. A suite is useful as long as its verdict arrives fast enough to stay tied to the change that triggered it, and as long as a red test really means "something is broken." Past about ten minutes, people switch contexts. Past a few tolerated random failures, they stop reading. This lesson therefore deals with two quantities: execution time and false failure rate. Everything else follows from them.

Splitting the Run into Stages

Don't launch everything at the same time. Fast tests should deliver their verdict before the slow ones start, so that failures happen early and agents are freed up.