Pratik R published a 12-metric evaluation harness for production AI agents this week on Towards Data Science, drawn from what the author describes as more than 100 enterprise deployments. It is one practitioner's playbook rather than a canonical standard โ and that is exactly why it's worth surfacing: the thresholds are concrete enough to lift, and the failure-mode framing names categories most benchmark suites still skip. The harness is grouped four ways: retrieval, generation, agent behavior, and production cost/latency. The origin story is more honest than most: a healthcare client's compliance officer asked "how do you know your agent isn't hallucinating patient symptoms" and the team had unit tests, integration tests, and a model that performed beautifully on the demo set โ but no way to measure hallucination rate, context faithfulness, or tool-selection accuracy on live traffic.
The concrete thresholds are the part to copy. Retrieval (4 metrics): context relevance above 0.85 on top-10 chunks, context recall above 0.90 on labeled benchmark queries, context precision above 0.80, retrieval latency under 200ms at p95. Generation (3): answer faithfulness above 0.95 against retrieved context, answer relevance above 0.90, hallucination rate below 2%. Agent (3): tool selection accuracy above 0.92, tool execution success above 0.98, multi-step coherence above 0.85. Production (2): cost under $0.05 per query typical, p99 end-to-end latency under 3 seconds. Most of these are scored by an LLM-as-judge evaluator โ which is the article's load-bearing caveat. LLM-as-judge has known reliability problems on the metrics that matter most, especially hallucination detection where the judge model and the agent model can share blind spots, and answer faithfulness where the judge can rate something faithful that a domain expert would not. The framework needs to be paired with human spot-checks at the threshold boundaries, not just blindly trusted.
The ecosystem read here lands against the editorial vacuum in agent evaluation. Frontier labs publish on saturated academic benchmarks (HELM, AgentBench, MMLU, GAIA) that test capability but not deployability; production teams have been quietly building in-house harnesses for two years without sharing what they look like. Pratik R's piece is a rare disclosure of an actual production harness's structure and thresholds, even if you discount the "100+ deployments" claim. The three failure patterns it names โ "we'll add evaluation after the MVP," "accuracy is enough," and "manual spot-checks are fine" โ match what every team building agents recognizes from their own experience. The 2% hallucination rate threshold is particularly load-bearing because most public benchmarks accept much higher rates implicitly by reporting only accuracy; for an agent answering customer questions or driving regulated workflows, 2% is the bar where shipping starts being defensible.
For builders: lift the four-category structure first (retrieval, generation, agent, production) โ the grouping is sound regardless of provenance. Lift the thresholds as starting points, then calibrate to your domain (healthcare needs hallucination near 0, customer-support can tolerate 3-5% if the agent escalates). Treat LLM-as-judge as the cheapest signal and pair it with periodic human review of borderline cases โ the article admits manual review breaks at 10K queries/day but doesn't fully address that LLM-as-judge can be confidently wrong about exactly the cases manual review would catch. The cost and latency targets are the boring half of the framework, and that's where most production failures actually live: an agent that hallucinates 1% of the time but costs $0.50 per query won't ship either. Pratik R's piece is at the TDS link; treat it as a starting reference, not a standard.