The Ratio
A weekly newsletter on reliability economics
The Number
1 of 57
Only 1 of 57 Technology companies in the benchmark has its reliability investment calibrated near-optimally — 29 under-invest and 27 over-invest.
Technology is the largest sector in the benchmark at 57 responses. 29 are classified as under-investing in reliability. 27 as over-investing. Only 1 is near-optimal.
Tech companies aren't failing to invest. They're failing to aim. A near-even split between under and over-investment, with one company in the middle, tells you the sector has a calibration problem, not a spending problem. Think of it like a factory floor where half the machines run too hot and the other half too cold, and exactly one thermostat works. The money is there. The dosage is wrong in both directions.
56 of 57 technology companies are either starving their reliability budget or drowning in it. One got it right.
This Week in Reliability
AI Agents Hit Production Reality
The first wave of autonomous AI agents is moving from demos to production infrastructure, forcing teams to confront new operational risks—agent observability, failure modes, and the cost of debugging multi-stage workflows—that existing SRE tooling wasn't built to handle.
Deep Reads
Wild AI-related reliability incidents are coming
Surfing Complexity · Primary evidence
Lorin Hochstein argues that AI agents are now capable of handling significant on-call work, citing Boris Tane's position that on-call is becoming theatre as agents take first-responder duties. The post explores how AI will change incident response workflows and the reliability challenges this introduces.
This is the reliability economics pivot point: when agents start changing production state autonomously, your incident taxonomy breaks. You're not budgeting for 'agent decided wrong' as a root cause category, but you will be.
Agent autonomy creates failure modes your runbooks don't cover.
Knowledge Graph as context for LLMs: demonstrating decisive RCA and faster production performance
Grafana Labs · Vendor response—agent context problem
Grafana tested how well AI agents can debug full-stack incidents and measured performance improvements when agents have access to Grafana Cloud's Knowledge Graph versus operating without that context. The post demonstrates that agent effectiveness in RCA depends heavily on contextual infrastructure data.
Here's the hidden agent tax: graph infrastructure. Agents need semantic context about your stack to avoid expensive hallucinations during incidents, which means new platform investment before agents deliver ROI. This shifts preventive spend forward in ways most roadmaps haven't accounted for.
Agent RCA quality depends on context infrastructure you don't have.
How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons
Grafana Labs · Infrastructure scaling for agent observability
Grafana shares capacity planning and load testing lessons for running Alloy as a centralized telemetry gateway handling enterprise-scale volumes—tens of millions of active series, terabytes of logs daily, and tens of thousands of trace spans per second.
Agents need telemetry pipelines that can handle both human-generated and agent-generated observability data at scale—double the cardinality problem.
Agent telemetry doubles your pipeline capacity problem overnight.
Automating root cause analysis at scale: Multi-signal correlation for cloud native incident response
CNCF Blog · Prerequisite for agent RCA
Atlassian describes the challenge of correlating hundreds of interconnected microservices across multiple regions during incidents, where telemetry volume overwhelms human operators and finding causal factors requires automated multi-signal correlation.
Multi-signal correlation is table stakes for agent-assisted RCA—this is the detection gap agents are supposed to close, but it requires investment in correlation infrastructure first.
Agents inherit your correlation debt before fixing it.
Safer Kubernetes rollouts with minReadySeconds
Gremlin · Agent safety prerequisite
The post explains how Kubernetes' minReadySeconds parameter makes new pods prove they can stay ready before a rollout proceeds, and provides guidance on setting it and identifying workloads that lack this safety control.
When agents start triggering rollouts, this config line becomes a reliability guardrail—the difference between an agent-driven deployment that self-heals and one that cascades.
Agent-driven rollouts expose every missing config guardrail.
Tough days at GitHub, a continuing series
Surfing Complexity · Real-world agent operational risk signal
GitHub experienced another major incident lasting nearly eight hours, with a public writeup posted unusually quickly. The post is the latest in Hochstein's ongoing series analyzing GitHub's operational challenges.
GitHub's incident cadence is a preview: as more companies deploy agents that touch production, expect your incident frequency to spike before it drops—agent learning curves cost MTTR budget.
Agent adoption raises incident frequency before it lowers it.
Nobl9 application (1.194.3) release notes
Nobl9 · Vendor product-ization of agent workflows
Nobl9 released Nobl9 Assist, an in-context AI assistant that answers plain-language questions about SLO burn, budget drain, and feature configuration using live organizational data. It can draft and apply configuration changes with explicit user approval.
In-product AI assistants mark the shift from 'agents as tools' to 'agents as coworkers'—you're now budgeting for agent UX, agent training data pipelines, and agent approval workflows inside every platform purchase.
Agent interfaces become a new platform buying criterion.
The Crowd Favorite
-
The Robots - 2009 Remaster - Kraftwerk ↗ — 2009 Remaster - Kraftwerk — Logging what an agent did without why it did it leaves you unauditable after the incident.
-
Howling - Âme Remix - Howling, RY X, Frank Wiedemann, Âme ↗ — Âme Remix - Howling, RY X, Frank Wiedemann, Âme — An agent that never throws an error can degrade output quality for weeks undetected.
-
HALCYON + ON + ON - Orbital ↗ — Orbital — One request fans out into forty tool calls. Untraced, the cost lands on nobody's budget.
-
Aaron - Paul Kalkbrenner ↗ — Paul Kalkbrenner — Nondeterministic runs can't be replayed, so your post-incident timeline is reconstruction, not evidence.
-
Morgens in Odonien - Dirk Sid Eno ↗ — Dirk Sid Eno — Agents keep the broadest permission they were ever granted. Scope creep is now silent.
Five blind spots in autonomous operations
The Challenger — Career Spotlight
Mid-size payments platform. 40-person platform org. Starting prevention-to-firefighting ratio: 0.71. Eleven agents running in production, and nobody could say what any of them had done.
Twelve weeks. Tooling came second. First they made every agent action emit a decision record: input, tool call, permission used, outcome. Then they cut agent scope back to what those records justified.
Ending ratio: 1.34. Agent-caused incidents had been roughly a third of the total. Nobody had attributed them there. Nothing was added to the budget.
You cannot rebalance what you cannot attribute
The Ratio is a weekly newsletter by Florian Hoeppner.
Take the assessment → reliabilityeconomics.com/benchmark
Reply to this email with your take.