Enterprise AI faces evaluation gap as agents gain autonomy

Enterprise AI faces evaluation gap as agents gain autonomy">

Enterprise AI teams are giving agents more freedom at the same moment their confidence in automated testing is collapsing. Half of enterprises have deployed an AI agent or LLM feature that passed internal evaluations and yet still caused a customer-facing failure — one in four more than once — according to the June 2026 VB Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees. The sample is self-selected rather than a probability sample, so the findings should be read as directional, not precise.

Enterprises are not responding by slowing automation: 66% of respondents already permit some production deployment without human review or are building systems intended to do so within the next 12 months. Only 5% say they fully trust the automated evaluations that would make those release decisions. That mismatch is the evaluation gap: the autonomy ceiling is rising faster than the assurance beneath it.

It also fits a broader thesis that will be explored at VB Transform 2026: enterprises ship agents first, while the control layers around identity, evaluation, cost, context and orchestration are arriving later. The next year will be a retrofit cycle, with buyers shifting budget toward the systems that make agentic deployments governable and dependable. Why a passing evaluation is not a working agent Traditional software testing usually asks whether a defined input produces an expected output.

Agent testing is harder because the system may choose its own sequence of steps, call tools, retrieve data, alter state and respond differently from one run to the next. An agent can make several individually plausible decisions and still reach the wrong result. It may retrieve the correct account but update the wrong field.

It may draft a valid refund request but send it without approval. The survey shows enterprises already recognize this limitation. The most common reason for distrusting automated evaluation is poor alignment with real-world outcomes, cited by 29% of respondents.

Bias or inconsistency follows at 21%, lack of explainability at 18%, and data leakage or privacy concerns at 17%. That hierarchy matters. Enterprises are saying the score often does not predict what happens when a customer, employee or business process encounters the agent in production — not that automated scoring is too slow or expensive.

NIST makes a similar point in its Generative AI Profile: measurements gathered in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context and operating conditions. Its guidance calls for field testing, post-deployment monitoring and clear processes for escalating failures. Capability is not consistency A single successful run proves that an agent can complete a task.