The enterprise AI industry faces a math problem. Cisco data shows 85% of enterprises are piloting AI agents, but only 5% have shipped them to production. At VB Transform 2026 on Tuesday, Bryan Silverthorn, Director of AGI Autonomy at Amazon, explained why that gap persists and why the answer isn't better benchmarks. Silverthorn, who joined Amazon through its acquisition of Adept AI and now leads multimodal agent training inside the company's AGI lab, argued that reliability must be broken into four distinct dimensions: consistency, robustness, predictability, and safety, a framework he credits to research from Princeton. These dimensions are often tangled together in most evaluations, he said.
Why AI agents pass internal evals but fail real customers in production
The framework matters because agents routinely ace internal evaluations and then collapse in the wild. Silverthorn described a customer that deployed an agent for software QA involving serial number extraction from screens. It worked flawlessly for two months, then began intermittently reading wrong numbers. The culprit: the underlying vision encoder behaved differently depending on where the serial number appeared on screen, and a software change imperceptible to humans triggered the failure. The lesson, Silverthorn said, is about measurement, not just models. Models have to be better, but the deeper takeaway is that teams need to identify their dimensions of variability and match measurement rigor to the stakes of the application. VentureBeat's own proprietary research, presented before the session, reinforces the point: half of surveyed companies shipped agents that passed internal evals but failed real customers, and enterprises overwhelmingly track uptime while ignoring accuracy. A related finding underscored how few guardrails exist: most enterprises default to the model makers' own evaluations, leaving their testing strategy a coin flip between trusting the vendor and trusting nothing.
Sponsored Protocol
Inside Amazon's 'intern' framework for managing autonomous AI agents
Silverthorn's most memorable prescription was cultural, not technical. Inside Amazon's AGI lab, researchers literally call their agents 'interns', as in 'I'll have my intern talk to your intern.' The joke carries a serious operational philosophy. Agents, like interns, are powerful but occasionally clueless, capable of amazing work and spectacular derailment. Managing them requires management skills rather than software skills: asking what could go wrong, adding backups and undo capabilities, and consciously deciding what risk you can accept. You can ask the intern: 'Hey, what might you do wrong here? How might you mitigate your negative outcomes?' he said. Amazon's lab has embraced that trade-off, accepting agents occasionally running the wrong experiment in exchange for research velocity, including one agent running experiments around the clock on its own high-level research plan.
Sponsored Protocol
What enterprise leaders should do before deploying agents at scale
Silverthorn was candid about the limits of today's technology. Self-improving AI remains a loaded term; Amazon uses AI to improve its models constantly, but fully autonomous self-improvement is distant. Computer use remains a core focus of his lab, with a commercial trucking customer already using browser automation to stitch together warranty claims across fragmented systems. He stressed that no future agent will rely on computer use alone; it will work alongside MCP, APIs, and other tools to complete end-to-end workflows. LLM-as-judge techniques, while promising, are just one of several strategies for aligning agent capability with acceptable risk.
Sponsored Protocol
For enterprises stuck in pilot purgatory, the path forward starts with a mindset shift: stop asking whether your agent can do something impressive once, and start asking whether it can do it correctly a thousand times in a row. In other words, the enterprises that escape the 85% ceiling won't be the ones with the smartest agents; they'll be the ones with the best managers. For more on AI agent safety, see our article on OpenAI GPT-Red. For an example of risk management in chatbots, check Meta and teen safety. An authoritative external analysis is available on Wikipedia on AI in industry.
Sponsored Protocol
