Enterprise AI Agent Evaluations Out of Sync: Half of Deployed Agents Caused Customer Failures After Passing

VentureBeat Pulse Research surveyed 157 enterprises with 100+ employees, revealing a severe reality gap in AI agent evaluations. 50% deployed AI agents or LLM features in the past year that passed internal evaluations but caused customer failures, with 25% experiencing multiple incidents. Only 5% fully trust current automated evaluations; the most cited limitation is poor alignment between evaluations and real-world outcomes (29%). The evaluation tool market is fragmented: top tools are model providers' native eval tools (17%) and having no dedicated tool (17%), while independent evaluation platforms are not yet standard. For production monitoring, only 23% check output quality in real-time, and 51% only monitor if agents run normally. Despite this, 66% either already allow (34%) or are actively building (33%) fully automated, zero-human deployment within 12 months.