The bill is a cardinality problem, not an observability problem

GitLab's infrastructure team once described its own monitoring stack, in a public issue, as "one of the least reliable services in our stack".^[1] Each Rails process exported over 70,000 metrics, more than 8 MB of text, four times a minute, and nine of its ten highest-cardinality metrics were histograms.

Cloudflare sits at the other end of the scale: 916 Prometheus servers holding around 4.9 billion time series, the largest about 30 million each.^[2] What monitoring costs, in memory and in money, is set less by how much you watch than by how many combinations you ask it to count. That number is cardinality, and most teams meet it for the first time on an invoice.

One metric, one label at a time

A Prometheus metric isn't one time series. It is one series for every distinct combination of its label values. Cloudflare's definition is a single line: "Cardinality is the number of unique combinations of all labels."^[2] So each label you add multiplies the count rather than adding to it.

Take the most common metric in any HTTP service, a latency histogram, with ten bucket boundaries. Before it has a single label it is already 13 series: eleven buckets (your ten and +Inf), a _sum and a _count. Then the labels arrive, each one a reasonable request at the time:

Every row up to pod is defensible. The last one is a single line in a pull request:

- requestDuration.labels(status, route, pod).observe(seconds)
+ requestDuration.labels(status, route, pod, user.id).observe(seconds)

That 249.6 million is an upper bound. Not every user calls every route, and not every route returns every status, so the real count is the number of combinations your traffic actually produces. It only has to be a small fraction of the bound to outweigh everything else you store.

Two things make it worse than the table suggests. pod changes on every rollout, so a deploy doesn't replace 31,200 series; it creates 31,200 new ones, and Prometheus keeps a series it has scraped even once in memory for one to three hours.^[2] And label values aren't always short. Cloudflare's example is an errors_total counter labelled with the error message, where every new file path or IP address in the message is a new series.^[2]

If you need to explain this to someone who doesn't write PromQL, a shop's sales are close enough. Count them by day and you have 365 rows a year. Count them by day and product, across 2,000 products, and it is 730,000. Add the customer and most rows hold a single sale that will never happen again. Each label is another column in that table, each series is a row, and the bill is per row.

Here is the same multiplication with your numbers in it. Click a label to take it out, change how many values it has, and send the link to whoever is about to add one; the numbers travel with it.

13×status×route×pod×user_id

= 31,200 series

Turning the lights off

When the bill arrives, the usual response is to turn the lights off: stop scraping a cluster, drop a namespace, switch off logs for a service. The bill falls. So does the visibility, on whichever system pages you next, and the label that caused the bill is still in the code, waiting for the lights to come back on.

It is also the wrong cut. What a system's monitoring costs is rarely spread evenly across it. In GitLab's case nine of the ten largest metrics were histograms, and the worst had 400,000 combinations on its own.^[1] Switching off a whole service throws away the cheap series that tell you it's up along with the few expensive ones that made the bill.

A fuse, not a diet

Prometheus has an automatic version of turning the lights off. sample_limit caps the number of series a scrape may return, and above it the entire scrape is treated as failed.^[3,4] Cloudflare's summary is blunter: "Prometheus won't scrape anything at all."^[2] The target goes dark, quite literally: its up metric drops to 0, as if it were down.^[3]

As a fuse, that is worth having. Without a limit, a single deploy that adds user_id can take a Prometheus server past its memory, and then every service loses its monitoring, not only the one that misbehaved. Cloudflare calls its limits "the last line of defense" against exactly that, and backs them with a CI check that its Prometheus servers have room before anyone's limit is raised.^[2]

As a way to reduce cardinality, it is a poor one, because you don't choose what it cuts. Upstream, it cuts the whole target. Cloudflare patched Prometheus to keep the series it already stores and refuse only new ones,^[2] which is kinder, but it still drops whatever is new, and during an incident what is new is often the signal: a status code you have never returned before, an error you have never seen. The limit also leaves the cause where it was. The label is still in the code, and the next deploy hits the same limit.

So set the limit well above normal, treat it as a fuse, and alert when it trips. Prometheus counts every scrape it rejects in prometheus_target_scrapes_exceeded_sample_limit_total. Then cut cardinality on purpose.

Cutting on purpose

To overcome this without turning anything off:

1. Bound every label before it merges. A label's values should come from a list you could write down: status codes, route templates, regions. User IDs, request IDs, raw URLs, error messages and timestamps can't be written down. Label with the route template (/users/:id), not the path (/users/8125).
2. Move per-request detail to traces and logs. They are built for values that occur once. If you need to get from a slow bucket to a request that landed in it, exemplars attach a trace ID to a histogram sample without making it a label, where your stack supports them (in Prometheus they sit behind --enable-feature=exemplar-storage).
3. Use fewer histogram buckets. GitLab's proposal was to cut most histograms to three to five buckets, set at the thresholds anyone actually alerts on.^[1] If your objective is "under 300 ms", that is the boundary you need.
4. Find the few series that cost the most, and drop those by name. metric_relabel_configs with action: labeldrop or action: drop removes a named label or metric at scrape time, before it is stored. Unlike a limit, you decide what goes.

Finding them doesn't need a vendor. Prometheus reports it about itself:^[5]

# the ten metrics with the most series
topk(10, count by (__name__) ({__name__=~".+"}))

# how many values one label has on one metric
count(count by (route) (http_request_duration_seconds_bucket))

The first query touches every series on the server, so run it once rather than putting it on a dashboard. The same breakdown, by metric name and by label pair, is on the TSDB status page and at /api/v1/status/tsdb. For Loki, logcli series '{}' --analyze-labels does the same for log streams.

I built Cardinal to do this without the queries. It is open source and runs in the browser: series by job, metric and label, log streams by service and label, a check of where each one is used, and the relabel, Alloy or Adaptive Telemetry rules to drop what you choose.

Making it reviewable

Cardinality doesn't show up in a diff, which is how user_id gets past code review. One line in the pull request template is enough to make it visible:

- [ ] New metric labels: name, most values it can take, and why it can't live in a trace or a log

Before adding a label, say its maximum number of values out loud, then multiply. If you can't say the number, it doesn't belong in a label.