Writing

Right-sizing telemetry budgets, not just collecting everything

Over the next year, I expect more enterprise observability initiatives to move from collecting everything to negotiating telemetry budgets by user journey and service tier. It is a small shift in language but a large change in practice. Collecting everything has been the default for a long time because storage was cheap enough and it felt safer than missing something. That default is now being questioned, not only because data volume keeps growing, but because the operational value of most of that data is unclear.

The phrase "negotiating telemetry budgets" sounds corporate, but it points at a real engineering decision. For each user journey—login, checkout, search, playback, whatever matters for the product—someone has to ask what telemetry would let an on-call engineer answer the first useful question when something breaks. The answer is rarely "everything at full resolution." Some journeys justify verbose tracing; others need only a few key metrics and a small set of logs.

Service tier adds another dimension. A tier with tighter SLOs might justify request-level traces, while a lower-risk tier might need only basic health telemetry and error counts. This is not about treating some users as less important. It is about matching the cost of observation to the operational risk and the questions you are likely to ask.

Cisco IT offers a concrete case. They cut endpoint telemetry volume by 75% while improving troubleshooting and resilience in Splunk Observability Cloud. The mechanism was not random downsampling. It was deciding what telemetry is genuinely required to answer operational questions, much like pruning unused code paths keeps a codebase navigable.

That distinction matters. Random downsampling treats telemetry as a statistical sample: you keep some fraction of everything and hope it is representative. The Cisco approach is closer to editing. You keep the records that map to specific questions about user journeys and service tiers, and you stop paying to store the rest. What is notable is that the reduction did not come at the expense of visibility. The team reported better troubleshooting and resilience, which suggests the previous volume included a lot of telemetry that was never being queried.

Pruning unused code paths keeps a codebase navigable because developers can find the parts that matter. Telemetry is the same. A dashboard with a hundred panels is less useful than ten panels that answer the questions the on-call rotation actually asks. When every service emits the same defaults, the signal disappears into the volume.

The practical implication is twofold. First, lower ingest cost and less noise. Second, and more importantly, shorter resolution times because responders are not sifting through irrelevant events. The real payoff is not the storage bill; it is reducing the time between an incident starting and someone understanding it.

I will be watching teams that pair right-sized telemetry with explicit retention and cardinality policies. These policies should be published and reviewed with the same discipline as SLOs. That means they are not set once and forgotten. They are revisited when services change, when questions change, and when people notice that some high-volume telemetry has not been used to resolve an incident in months. Retention and cardinality determine how long you keep each class of data and how many distinct label values you allow. Without explicit policies, teams drift toward defaults that are either too generous or too restrictive.

The signal to watch is whether MTTR improvements follow the data reduction, or if teams just trade one blind spot for another. A lower ingest bill is easy to measure. The harder question is whether responders can still see what they need when a complex incident crosses service boundaries. Some blind spots only show up later, in the gap between what was kept and what turned out to matter. If MTTR improves, the team removed noise without removing signal. If it stays flat or gets worse, the team may have removed low-volume but high-value data, or made it harder to correlate across services.

Originally covered at blogs.cisco.com ↗