Suite Speed and Reliability

Nobody runs an eighteen-minute suite before pushing. A suite that fails one run in six for varying reasons stops being read: the team reruns it, and if it passes, they merge. In both cases, the suite has stopped doing its job, and not because it contains bad tests — but because it has become too expensive or too noisy to deserve anyone's attention.

These two kinds of degradation creep in gradually and are handled the same way: by measuring. Nobody correctly guesses where a test suite's time goes, and nobody guesses the cause of a flaky test. Both can be observed, using tools that are already in your project. This lesson therefore starts by reading the data that Surefire and Gradle already produce, then moves on to parallelization, the real causes of flakiness, and what code coverage actually measures.

Measure Before Optimizing

Surefire writes one XML file per test class in target/surefire-reports, with the duration of each method. Sorting by descending duration is enough to identify the culprits.

# The 15 slowest tests, across all classes
grep -h '<testcase' target/surefire-reports/*.xml \
  | sed -E 's/.*name="([^"]+)".*classname="([^"]+)".*time="([^"]+)".*/\3 \2#\1/' \
  | sort -rn | head -15