A practical guide to evaluating LLM and agent systems
Metrics, datasets, human evals, and the automated LLM judge frameworks we use to measure real reliability.
The Three Layers of Agent Evaluation
You cannot improve what you cannot reliably score. In early-stage development, teams rely on vibes: an engineer tries three prompts in a playground, nods approvingly, and pushes to production. In enterprise deployments, this approach leads to rapid regressions where fixing one customer complaint breaks five others.
To establish repeatable reliability across hundreds of thousands of sessions, we structure agent evaluation into three rigorous operational tiers:
1. Deterministic Invariant Checks: Non-LLM unit assertions that verify tool argument schemas, valid UUIDs, regex boundary rules, and non-empty responses in milliseconds. 2. Isolated Synthetic LLM Judges: Automated multi-turn test suites that run recorded customer transcripts through dedicated evaluation models using explicit binary rubrics. 3. Blinded Human Review Panels: Weekly sampled reviews conducted by senior staff and domain experts, comparing agent decisions against gold-standard policy documents without knowing which prompt variant produced the output.
Never let the model evaluate its own output in the same context window. Always use an isolated judge agent with blinded instructions and zero access to generation scratchpads.
Calibrated Binary Rubrics Over Vague Likert Scales
Asking an LLM judge to 'rate this response from 1 to 5 for helpfulness' produces noisy, uncalibrated scores that drift by 15-20% across model patch releases. Subjective ratings conflate length with quality and flatter models that use polite, verbose filler.
Instead, we decompose quality into deterministic binary criteria (True/False) evaluated against explicit ground-truth contracts. Each question has a crisp definition with zero ambiguity:
• Did the agent answer the customer's question without re-asking for information already present in context? (True/False) • Did the agent adhere to the strict no-refund policy for digital subscriptions without fabricating exceptions? (True/False) • Were all tool parameter values verified against the retrieved customer CRM context? (True/False) • Did the agent avoid mentioning competitor products when responding to comparison queries? (True/False)
Aggregating 15 binary checks produces an objective, repeatable score that correlates with human engineering audits at over 94% statistical agreement.
// Calibrated binary eval rubric parser
import { z } from "zod";
export const EvalChecklistSchema = z.object({
checks: z.array(z.object({
id: z.string(),
criterion: z.string(),
passed: z.boolean(),
rationale: z.string().min(10),
})),
criticalFail: z.boolean(),
aggregateScore: z.number().min(0).max(100),
});
export type EvalChecklist = z.infer<typeof EvalChecklistSchema>;
export function calculateWeightedEvalScore(evalResult: EvalChecklist): number {
if (evalResult.criticalFail) return 0;
const passedCount = evalResult.checks.filter(c => c.passed).length;
return Math.round((passedCount / evalResult.checks.length) * 100);
}Automated Evaluation Gating in CI/CD Pipelines
Every prompt change, temperature adjustment, or SDK upgrade must be treated with the same engineering rigor as a core database migration. We gate every pull request behind an automated regression pipeline that executes against 500+ curated historical transcripts.
If overall tool parameter accuracy drops by more than 0.5%, or if any critical safety check fails on a known edge case, the GitHub Action blocks merge immediately and annotates the PR with side-by-side diffs showing the regression.
Hard Negative Mining from Production Outages
Static benchmarks become stale within weeks as user behavior evolves. We maintain a continuous feedback loop called Hard Negative Mining.
Whenever a production conversation results in user correction, early session abandonment, or human agent escalation, the session transcript is automatically sanitized of PII, tagged with failure tags, and committed as a new test fixture. Our evaluation test harness grows stronger every single day because real-world failures become permanent automated guards.
The best evaluation benchmark is not an academic dataset like MMLU. It is your own repository of the last 200 real customer conversations that went off the rails.
Controlling Model Variance in Judge Generations
LLM judges are themselves stochastic models. If you run the exact same eval suite twice, you will occasionally get slight score deviations due to floating-point nondeterminism and temperature variations.
To ensure scientific reproducibility, we lock judge model temperatures to 0.0, use greedy decoding, enforce structured JSON schema outputs, and run edge cases through a 3-way consensus vote across independent model families whenever a critical score is on the boundary.
Rohit Anand
Engineering Leader & Builder building enterprise platforms, Voice AI, and Agentic systems. Dedicated to evolutionary architecture, team judgment, and delivery excellence.