Two weeks ago I wrote about the failure modes that hide when agents move from demo to production. Top of the list: the one where the agent quietly produces something plausible instead of saying “I don’t know.”
Datadog just put a number on it. One in twenty requests in production AI systems fails silently — the output looks right, the system keeps running, nothing trips an alarm. By engineering standards, a 5% silent failure rate is astronomical.
The number sounds smaller than it is. If your agent runs a thousand decisions a day, fifty of those are wrong in ways you can’t see from looking at the output. The errors don’t show up in logs because nothing failed. They show up downstream, days or weeks later, when somebody notices a number that doesn’t add up or a customer complaint that traces back to a recommendation you can’t explain.
The catastrophic version is in regulated industries. A 5% silent failure rate inside a clinical trial workflow is regulatory exposure, not a quality issue. The same rate in financial reconciliation is an audit finding. In environments where someone has to sign off on the work, the AI output isn’t usable at all until the eval layer can catch those failures before they propagate.
Most teams haven’t built it. Datadog’s broader finding: fewer than 40% of production agents have automated evals running on every prompt change. The 5% is being discovered by customers, not by the team running the system.
What every ROI deck skips: the eval and observability layer isn’t optional. If your agent is in production without it, you don’t have a production system. You have an unsupervised intern with API access, and you’ll find out what they got wrong from somebody downstream.
If you’re running anything in production right now, pick one agent and ask the team how they’d know if it was wrong 5% of the time. If the answer is some version of “we’d hear about it,” you have your next project.