What Prometheus metrics a service mesh should expose

A service mesh sits in the path of every service-to-service call in your system. That position makes it a natural place to collect observability data. But 'expose Prometheus metrics' covers a wide range. A mesh can expose dozens of counters that tell you very little, or a small set of metrics that answer the questions you actually have at 2 a.m. This article covers the metrics that matter and why.

Request-level metrics ¶

The four metrics that answer most production questions are: request rate (requests per second, broken down by source service, destination service, and HTTP method), error rate (the fraction of requests returning 5xx or connection errors), latency histogram (p50, p95, p99 broken down by the same dimensions), and in-flight request count. These four, with the right label dimensions, let you identify which service pair is having a problem and whether it is a latency issue or an error rate issue.

Label cardinality is the main risk. If you label by individual pod or instance rather than by service, the number of time series explodes as you scale. Label by service name and, if you need it, by a stable deployment identifier. Not by pod UID or IP address.

Certificate and identity metrics ¶

Certificate expiry is the metric most teams add too late. A counter for certificates expiring within the next N hours, broken down by workload, lets you alert before a rotation failure causes a connection drop. Tuft Mesh Vault exposes tmv_certificate_expiry_seconds as a gauge per registered workload. A simple alert on values below 3600 (one hour) catches most rotation problems before they affect traffic.

Policy decision metrics ¶

Policy decision counters. How many requests were allowed versus denied, broken down by rule. Are useful for two things: verifying that a newly deployed policy is behaving as expected, and detecting unexpected traffic patterns (a service calling another service it should not be calling, for example). Tuft Mesh Vault exposes tmv_policy_decisions_total with allow/deny and rule_name labels.

The full metrics reference, including all label dimensions and example PromQL queries, is in the Tuft Mesh Vault docs under 'Observability'. If you are building a Grafana dashboard from scratch, the docs include a starter JSON export.