The Evaluation Gap: Why Companies Trust AI Agents More Than Their Own Tests

The Trust Paradox in the Age of Autonomous AI Agents

The numbers may sound abstract at first glance, but they strike at the heart of one of the most pressing challenges in modern AI development: every second company has, over the past twelve months, deployed an AI agent or LLM feature into production that had passed all internal evaluations – only to fail in a customer-facing situation. At the same time, two thirds of the organisations surveyed plan to use precisely these insufficiently trustworthy evaluations as the sole release criterion going forward – entirely without human oversight. This paradox, documented in a recent survey of 157 companies, marks a critical inflection point in enterprise AI.

This is not a fringe problem for technical teams. It is a structural risk that grows exponentially in significance as agent autonomy increases. Dr. Maik Bunzel, founder and managing director of mabucon.eu, frames this finding pragmatically: "The real problem is not that evaluations are missing – most companies conduct them. The problem is that the metrics are not aligned with what matters in the real world: customer outcome, process stability, business outcome."

What the "Evaluation Gap" Actually Means

The term Evaluation Gap describes the distance between two developments moving in opposite directions: the autonomy companies grant their AI agents continues to grow, while confidence in the tests meant to safeguard that autonomy stagnates or even declines. The result is a widening gap – and caught in the middle are customers, business processes, and reputation.

• 50 percent of companies that conduct evaluations have released an agent at least once that subsequently failed in a customer-facing situation.
• 24 percent have experienced this failure more than once – an indication that structural, rather than coincidental, causes are at play.
• Only 5 percent fully trust automated evaluations.
• 29 percent cite the most common weakness as: evaluation results do not reflect real-world outcomes.

These figures illustrate that the widely adopted approach – testing an agent against a test dataset or benchmark set and deploying it upon passing – is fundamentally inadequate. Benchmarks measure what appears in benchmarks. What customers actually experience is an entirely different dimension.

The Autonomy Escalation: Speed Without Safeguards

Particularly revealing is the direction in which the surveyed companies are heading. Despite widespread distrust of automated evaluations:

• 34 percent already permit fully automated deployments without human approval – at least for agents classified as "low risk".
• 33 percent are actively building towards exactly this infrastructure, with the goal of reaching that point within twelve months.
• Only 22 percent rule out fully automated deployment for the foreseeable future.

What makes these numbers particularly alarming: larger companies are further along this path than smaller ones. The widespread assumption that regulated or resource-rich organizations act more cautiously and rely on human oversight for longer is not confirmed by this data. On the contrary – larger companies have more frequently experienced evaluation failures in production, yet are pushing harder to automate release processes.

Autonomy arrives faster than the safeguards. This sentence precisely describes the mechanism by which one-off errors become systemic risks.

Fragmented Tool Landscape: Nobody Has Solved the Problem

Another finding underscores the immaturity of the field: most companies rely on the native evaluation tools provided by their respective model vendors – or have no dedicated evaluation tooling at all. Both groups account for around 17 percent each. Only approximately one quarter of companies conduct real-time quality checks on live production traffic.

This fragmentation is no coincidence. It reflects the state of a market segment that is still in its formative phase. LLM-Observability, Agent-Tracing, production monitoring – all of these disciplines exist, but are neither standardized nor broadly established. Companies are building on a foundation whose load-bearing capacity they themselves doubt.

Drawing on practical experience in agent implementation, Dr. Maik Bunzel von mabucon.eu emphasizes that this particular point is frequently underestimated: "Many companies invest considerably in the development and training of their agents, but barely in the infrastructure that ensures these agents perform as intended in day-to-day operation. This comes back to bite them at the latest when an agent fails during a customer interaction or a critical workflow."

Why "Low-Risk" Is a Deceptive Category

One conceptual problem deserves particular attention: the category of "low-risk agents," for which fully automated deployments are already permitted by a majority of companies. What qualifies as low-risk is defined in practice by the teams building the agent – and this definition is rarely systematically reviewed or tied to real-world harm scenarios.

An agent that classifies emails appears harmless. However, if it misclassifies at a volume of thousands of transactions per day and those misclassifications trigger downstream processes, the cumulative risk is considerable. Autonomy scales errors – that is the real danger lurking behind the survey results.

• Evaluations test isolated capabilities, not systemic effects.
• "Passed" tests measure performance against known scenarios – not against the unpredictability of real-world inputs.
• Without production monitoring, it remains unclear whether an agent operates after deployment the same way it did during the evaluation phase.

What Companies Can Do Right Now

The findings do not call for a return to manual processes or a slowdown in AI adoption. They call for a different kind of maturation: the infrastructure with which autonomy can be scaled responsibly. For companies integrating AI agents into operational processes, concrete areas for action emerge:

• Connect evaluation to production reality: Offline benchmarks must be supplemented by continuous monitoring of real production data. Shadow deployments, A/B tests, and automated anomaly detection on live traffic are not luxury features — they are operational necessities.
• Formalise risk classifications: What qualifies as "low risk" requires an explicit, traceable definition — anchored in damage scenarios, case volume, and process criticality, not subjective assessment.
• Place Human-in-the-Loop deliberately: Humans do not need to be everywhere, but they must be present wherever evaluations are demonstrably weak. This demands transparency about where evaluations fail — which in turn requires explainability within the evaluation process itself.
• Diversify tooling: Dependence on native model-provider Evals is a structural risk. Provider-independent evaluation and monitoring solutions enable more consistent standards across different models and agents.

Outlook: The next maturity level of autonomous systems

The Evaluation Gap is not an inevitable fate. It is the consequence of a specific choice: granting autonomy earlier than the systems designed to safeguard that autonomy are production-ready. This decision is often made under competitive pressure — deploy more slowly and you fall behind competitors, so the logic goes.

What this logic overlooks: an agent that repeatedly fails in production does not merely cause immediate harm. It erodes institutional trust in AI-driven processes as a whole — making future adoption harder, not easier. The organisations that invest now in robust evaluation and monitoring infrastructure are creating the foundation for agent autonomy to remain viable in the long term.

Dr. Maik Bunzel, founder and CEO of mabucon.eu, summarises the strategic implication: "We are at a point where the technical feasibility of AI agents is outpacing the organisational capability to govern them. The competitive advantage of the coming years lies not in deploying agents faster — but in deploying them in a way that reliably delivers what they were built to do." The Evaluation Gap is solvable. But it will not solve itself.