An LLM agent produces text, but a production workflow must produce typed output that a downstream step, or a customer's decision, can trust. The approach here is not to review an agent's prose and call it good. The agent is run against fixed inputs, and invariants are checked on its structured, schema-validated output. If the output violates an invariant, the sample fails. If enough samples fail, the change does not ship.
The whole system sits in core/evaluation, and it is built around one function: run_scenario. It seeds a real job through the real paths, real template keys, real JobTransition rows, a real claim, then runs the agent with stubbed tools, finishes the step, and applies the scenario's checks. Crucially, it never raises for agent failures. A bad run becomes an outcome, not an exception.
The checks are deliberately narrow and encoded, not prompted. Three examples from core/evaluation/checks.py.
assert_facts_grounded. Every valued fact in the verified intel must cite a URL that was actually retrieved, and its quote must appear verbatim, whitespace- and case-normalised, in that source's text. A fact that points at something never fetched, or whose quote does not exist, fails.assert_revenue_not_on_file. The absence side of the same invariant. When no retrieved source mentions revenue, the revenue fact must be value=None. A hallucinated figure fails outright.assert_price_from_notes. Any price in the offer draft must appear in the operator's own pricing notes. A £14,000 that the operator never wrote down is a failure.These are universal invariants attached to a template in CHECKS_BY_TEMPLATE. synthesize always runs the grounding check. draft_offer always runs the price check. That mapping matters because it is exactly what a replay of a real job runs too.
The fixed scenarios live in core/evaluation/scenarios.py as an AgentScenario registry. Each one pins the agent, the template, the requirements, the seeded inputs, and the checks, plus a max_failure_rate, set to 0.01 for these examples.
The fixtures are committed and anonymised: a fictional "Acme Widgets Ltd" with a Companies House filing and a crawled "About" page. synthesize.grounded gives the agent sources that do contain revenue, so the grounding check should pass. synthesize.missing_revenue gives it a filing with "No accounts filed" and a page that says "contact us for a quote", so any revenue figure is a hallucination and assert_revenue_not_on_file must fail it. synthesize.after_rejection re-runs the same scenario with feedback attached, "the revenue figure was wrong last time", to prove a rejection is actually learned from. draft_offer.price_from_notes tests that a price is lifted from pricing notes, not invented.
Real-world fixtures are a different beast and live in EVAL_FIXTURE_DIR, deliberately never committed to version control. Anonymised data is checked in so anyone can reproduce a failure. Real customer data stays out of the repository.
Synthetic cases are necessary but not sufficient. The corpus answers a sharper question: would the agent have passed on the jobs a human already decided?
scenario_from_job rebuilds exactly what a real job's first attempt saw, selected by time, not the run's current state, which may have drifted since. It reconstructs the requirements and upstream outputs as of the first claim, and attaches the universal checks for that template. replay_corpus gathers live jobs with at least one approval decision, newest first, using the first decision because later attempts were shaped by operator feedback. A case whose human verdict was "rejected" carries its rejection reason as the feedback on replay. "Approved" cases carry none.
This is the important part. The agent is judged against what production actually saw, not a curated example. Every corpus sample is stamped with source_job_id and case_verdict so the audit trail points back at a specific real decision.
The management commands escalate from "does one step work" to "does the whole workflow survive a batch".
wf_step runs one step once, verbosely. --scenario picks a registered scenario. --job replays a real job id. --show-prompt dumps every prompt and response. With --interactive you can simulate rejections in a loop, up to three, feeding the rejection reason back as feedback each round.run_agent_batch runs a scenario or a replay corpus N times and gates on the failure rate. It writes an EvalSample per run, then reads the gate from the database, so samples written by Celery workers count too.wf_batch drives the entire workflow N times from a fixture, acting as the operator through the real services, approving and rejecting like a human would.wf_run is the interactive operator REPL: a full run driven by hand, with every approval, rejection, and submission attributed to a real operator user.Pass/fail is computed from a failure rate against a threshold, and the arithmetic is deliberately careful. In BatchSummary, transient samples are excluded from the rate. A rate-limited provider says nothing about your agent, so judged = samples - transient, and the rate is (check_failed + errors) / judged. Failures caused by the agent count against you. The weather does not.
But the exclusion has a limit. If more than --max-transient (default 0.2) of samples are transient, the batch is inconclusive. You cannot judge an agent on a provider outage, so run_agent_batch exits 2 instead of pretending to pass or fail. A rate above the threshold exits 1, so CI can gate a merge directly. The batch writes an EvalRun with every sample saved for audit.
samples: 100
passed: 98
check_failed: 1
transient: 1 (excluded from the rate)
failure rate: 1.0% (threshold 1.0%)
gate: PASS
For corpus runs, rejected cases are excluded from the gate. They have no "right answer" to check. Only case_verdict="approved" samples are judged.
A customer never sees the first version of a step. They see the version that passed fixed invariants, that replayed hundreds of real historical decisions without regressing, and that a machine, not a reviewer's confidence, declared fit. That is the difference between shipping an agent and shipping a governed workflow. Every step is proven against a threshold before it ever touches a customer.
You can't ask an expert to write down what they know. Capture what they do.
Capturing expert judgement in a governed workflow, not a written manual, and turning craft into repeatable systems.
AI can do the legwork. It can't do your judgment.
Gathering and summarising information is now a commodity. The judgement layer is the real moat.