A single exchange with an AI agent can look flawless when scored in isolation and still point to a broken product. This gap is driving a shift in how enterprises evaluate agents: away from scoring individual traces toward comparing cohorts of users against a baseline. At VB Transform 2026, Harrison Chase, CEO of LangChain; Hui Zhang, CTO and co-founder of Conviva; and Emmanuel Turlay, director of engineering at CoreWeave, described that shift, along with a parallel move toward cheaper, narrower judge models.
Agent-as-judge has not replaced LLM-as-judge
According to Chase, the agent-as-judge approach has not supplanted llm-as-judge, which remains the default. The larger tension, Zhang noted, is between automated judging (whether by LLM or agent) and human review. “You have scalable but ungrounded”, Zhang said, referring to the difficulty of grounding automated evaluation. The industry must pick between two poisons: scalable but ungrounded automation, or unscalable human review.
Sponsored Protocol
Evaluation criteria as product spec
Chase framed evaluation criteria as a living specification: “Evals are the new PRD”. He warned against “eval paralysis”, where teams don't launch because the test suite isn't perfect. The best teams launch and iterate. Turlay agreed: chasing 100% coverage does not prevent bugs in production. Better to have broad, always-on monitoring to identify real failure classes, then build a targeted offline evaluation set.
Why scoring traces one at a time is a mistake
Zhang criticized the common method of sampling traces and scoring each in isolation. This approach misses signals that only appear when comparing cohorts of users against a baseline, a method he calls contrastive analysis. He gave a retail example: a single interaction for running shoes looked fine, but the clarification ratio (how many follow-up questions the agent asked) was three times higher than baseline, and the rate of purchases completed outside the conversation was five times higher. Neither number is visible from a single trace.
Sponsored Protocol
Sizing the judge to the job
Turlay recommended starting with the most capable model to prove a task is solvable, then scaling down. Once proven, teams can sample a fraction of traffic and move simple tasks like binary classification to smaller open source models. LangChain went further, fine-tuning a Qwen model to detect when a user perceives an error, achieving performance similar to Claude Sonnet with a 10 to 100x cost reduction.
LLM-as-judge does not eliminate the human in the loop
Both Turlay and Zhang agreed that a human must remain responsible for corner cases and legal liability. Chase added that human oversight is crucial for building trust and enabling system learning: “There has to be interactions in order for the system to learn”. Cheaper judge models and contrastive analysis are reshaping how companies validate AI agents, but the human factor remains irreplaceable.
Sponsored Protocol
The need for reliable evaluation is underscored by recent $3.7 billion in deepfake fraud losses and the discovery of worms hiding in AI development pipelines. For further reading, see Wikipedia on artificial intelligence.
