Agentic toolkit

Grading what an agent did

agent-evals takes agent trajectories from any source, matches them to scenarios written as data, runs deterministic graders and an optional LLM judge, reports pass rates with confidence intervals, compares against a baseline and decides a CI gate. It exists to make FleetAlert’s evals reusable, and its first job is grading FleetAlert’s live traces from the outside.

Read the source and decision record, the FleetAlert write-up, or the eval results FleetAlert publishes itself.

One spine, three checks on it

FleetAlert trace APIspans → one round eachClaude Messages APItool_use / tool_result turnsAny agentJSONL, one line per runAdapters → one Trajectory formatstatus ok · truncated · error: plumbing failures go to a sidecar, never the pass ratematch by field paths or scenario_idScenarios (YAML) × graders15 atomic, deterministic graders · shared check listsllm_judge: structured output, candidate framed as data,its cost recorded apart · non-gating until calibratedselfcheckknown-good must pass;emptied copies must fail(oracle and null)per-trial gradesStatisticsWilson 95% interval on every rate · noise floor ≈ 1/√nregression only when baseline and current intervalsdo not overlap · cost increase vs baselinecalibratejudge vs human labels:agreement, Cohen's kappa,at every thresholdheadline recomputed from rowsGate + reportper-scenario and overall thresholds · coverage (min_trials)exit 1 on failure · markdown to the job summary · baseline promotionGitHub Actionuses: mbgolden/agent-evalsfails the job on the gateFirst consumer: FleetAlert’s eight golden scenarios grading live rounds from the public demo.

Everything on the left is the path a trajectory takes. Everything on the right checks the harness rather than the agent: that the graders can fail, that the judge agrees with people, and that the whole thing runs in CI.

The five-minute version

01

Grade the trajectory, not just the answer

An agent's output is a sequence of decisions: what it read, what it called, what a guardrail refused, what it ended with. agent-evals takes that sequence from any source through an adapter (FleetAlert's trace spans, a Claude Messages API conversation, a JSONL log) into one trajectory format, and every grader is written once against it. FleetAlert's own in-repo evals read its spans directly; this makes the part after the agent's log format reusable.

02

Plumbing failures are not model failures

A run cut off at max_tokens, a round still in flight, a crashed harness: each has a status, lands in an errors sidecar in the results, and never enters the pass rate. The checklist this follows puts it plainly: anything that scores an infrastructure error as a wrong answer is contaminating the headline. A grader that itself cannot run (a missing field, no cost recorded) reports that, rather than failing the trial.

03

Scenarios are data, graders are atomic

A YAML file says what a correct run looks like and why, with a match on trajectory fields and a list of checks naming a grader and its parameters. Fifteen built-in graders each check one property: end state, evidence gathered before acting, no guardrail refusals, cost, latency, step and model-call budgets, text and numeric fields. Each ships with a known-good and a known-bad trajectory it must separate. Graders that read the end state are preferred over ones that dictate a path.

04

A judge you can trust, or know you cannot

The LLM judge scores a rubric with structured output, frames the candidate text as untrusted data, is told not to reward length, records its own token cost apart from the agent's, and is non-gating by default. The calibrate command runs it over human-labelled trajectories and reports agreement and Cohen's kappa at the configured threshold and every other one, with the disagreements listed. Below about 90% agreement the report says so, and the judge stays advisory.

05

Every number carries its uncertainty

Three passing trials is a 44% to 100% Wilson interval, and the report says so, along with the noise floor for the trial count. A regression is called only when the baseline's interval, whose trial count is stored, and the current one do not overlap. At three trials each that never fires, which is the honest answer; the absolute thresholds carry the gate at small n, and the regression flag earns its keep as trials grow. The headline is recomputed from the per-trial rows.

06

A gate, and a check on the gate

Per-scenario and overall thresholds, a coverage rule so a run that quietly skipped half the suite cannot pass, cost-increase limits, exit code 1 on failure, and a baseline command that refuses to promote a run that failed its own gate. The selfcheck command runs known-good trajectories through the suite (every gating check must pass) and their emptied copies (nothing may pass), catching a wrong field path or an always-true grader before a paid run.

07

First consumer: FleetAlert, graded from the outside

FleetAlert's eight golden scenarios fit in one file and grade rounds fetched from the live demo's public trace API. On the day of the snapshot, seven complete rounds passed every check: evidence before proposing, no refusals, cost under $0.15, a rationale citing a knowledge-base entry or a reading, the expected end state. Four scenarios had no rounds, because nobody had asked the demo to investigate those alerts since its last reset, and the gate failed on coverage rather than hiding it. A weekly workflow re-grades the live demo so the example stays honest.

Key decisions, from the record

ADR-0001
One trajectory format, with adapters. A new agent needs an adapter, not a harness. Status keeps plumbing out of the pass rate; numbers come from the agent's own token accounting, never estimated from string length.
ADR-0002
Scenarios as YAML; atomic, deterministic graders. Shared check lists, field paths into steps, labels that make a failing row read as the expectation. Both directions covered, so an always-act policy cannot score perfectly.
ADR-0003
The judge is non-gating until calibrated. Structured verdicts, the candidate as data, known negatives scored low by instruction, judge cost kept apart, and a calibration report before anyone promotes it.
ADR-0004
Intervals, non-overlap regressions, coverage, selfcheck. Wilson intervals on every rate, a stated noise floor, regressions only when intervals separate, min_trials in the gate, and an oracle/null check on the suite itself.

By the numbers

15
Built-in graders, each tested against a known-good and a known-bad trajectory
95%
Wilson interval on every pass rate, with the noise floor stated
7/7
Live FleetAlert rounds that passed the example suite on the first snapshot
4
Scenarios the gate flagged as uncovered that day, instead of passing silently
40
Tests, including the judge against a fake client and the CLI end to end
4
Architecture decision records

Built against a checklist, not a hunch

The design follows an eval-health checklist: infrastructure errors apart from model errors, outcomes over paths, multiple trials with intervals, a judge calibrated against human labels and tested on known negatives, and an oracle and a null baseline run through the pipeline before trusting it. Each of those is a feature here, not advice in a README.

It was built with Claude Code as a pair. The agent wrote the adapters, graders, statistics and CLI against the checklist; the judgment calls stayed with a person: which graders are allowed to gate, how conservative the regression rule should be at small trial counts, and that a coverage gap is a failure rather than a footnote.