Prometheus and Its Data Model

A Prometheus server that runs well for six months can become unusable in a single day because of one seemingly harmless line of code. Someone adds a user_id label to a request counter so they can "filter by customer." The next day, the server's memory usage has gone from 4 to 38 GiB, the main dashboard's queries time out, and the process ends up killed by the kernel. No alert fires, because the alerting system runs on the very server that just died.

This outage is the most common in the entire ecosystem, and it is entirely predictable once you understand what a time series is to Prometheus. It is not a named metric that you filter at read time, like a column in a SQL table: each distinct combination of label values creates an independent series, with its own in-memory structure and its own file on disk. Ten thousand customers multiplied by five endpoints multiplied by four status codes is not ten thousand rows of data; it is two hundred thousand series. Understanding this model before you write your first metric will save you from rebuilding your installation six months from now.