How zero-downtime certificate rotation actually works

Certificate rotation sounds straightforward until you think about what happens to in-flight connections during the switchover. A connection established with the old certificate is still open. The peer has validated the old certificate. If you revoke the old certificate the moment the new one is issued, that connection breaks. Zero-downtime rotation requires a period where both certificates are simultaneously valid. An overlap window. And some mechanism to ensure the new certificate has reached all peers before the old one expires.

The overlap window ¶

The overlap window is a period during which both the outgoing and incoming certificates are valid. The length of the window needs to be long enough for the new certificate to propagate to all sidecars in the mesh. In a small mesh with fast networking, a few seconds is enough. In a large mesh with hundreds of sidecars across slow network segments, you might need a minute or more.

Tuft Mesh Vault defaults to a 60-second overlap window. This is configurable down to 10 seconds if your mesh is small and your network is fast. Setting it too short is the most common cause of rotation-related connection drops.

Propagation barriers ¶

An overlap window alone is not sufficient if you do not know when propagation is complete. Without a propagation barrier, the CA advances the expiry timestamp of the old certificate on a fixed schedule, regardless of whether all sidecars have received the new one. If a sidecar is slow to acknowledge. Due to a network hiccup or a high load spike. It may still be presenting the old certificate when the CA marks it expired.

A propagation barrier polls each registered sidecar for acknowledgement of the new certificate before advancing the expiry. This is what version 0.14.3 of Tuft Mesh Vault fixed: a race condition where the barrier could, under specific timing, advance the expiry before all acknowledgements were collected. The fix adds a strict sequencing guarantee: expiry advancement only happens after all acknowledgements are received or a conservative fallback timeout is reached.

What this means for your rotation interval ¶

With a correct overlap window and a propagation barrier, you can rotate certificates frequently without connection drops. The practical lower bound on the rotation interval is the time it takes for the slowest sidecar in your mesh to acknowledge a new certificate, plus the overlap window, plus some margin. For most meshes this is well under five minutes, which means hourly rotation is achievable without any special configuration.

The cert rotation configuration reference is in the docs under 'PKI and identity'. If you hit a rotation issue in production, the audit log will have a record of every certificate event with timestamps. That is usually the fastest way to diagnose what happened.