[{"data":1,"prerenderedAt":490},["ShallowReactive",2],{"navigation":3,"/blog/how-we-test-every-step-and-workflow":96,"/blog/how-we-test-every-step-and-workflow-surround":479},[4,23,60,80],{"title":5,"path":6,"stem":7,"children":8,"icon":22},"Getting Started","/docs/getting-started","1.docs/1.getting-started/1.index",[9,12,17],{"title":10,"path":6,"stem":7,"icon":11},"Introduction","i-lucide-house",{"title":13,"path":14,"stem":15,"icon":16},"Core concepts","/docs/getting-started/concepts","1.docs/1.getting-started/2.concepts","i-lucide-lightbulb",{"title":18,"path":19,"stem":20,"icon":21},"Your first run","/docs/getting-started/first-run","1.docs/1.getting-started/3.first-run","i-lucide-play","i-lucide-rocket",{"title":24,"icon":25,"path":26,"stem":27,"children":28,"page":59},"Operator Guide","i-lucide-layout-dashboard","/docs/operator-guide","1.docs/2.operator-guide",[29,34,39,44,49,54],{"title":30,"path":31,"stem":32,"icon":33},"Runs","/docs/operator-guide/runs","1.docs/2.operator-guide/1.runs","i-lucide-git-branch",{"title":35,"path":36,"stem":37,"icon":38},"Targets","/docs/operator-guide/targets","1.docs/2.operator-guide/2.targets","i-lucide-building-2",{"title":40,"path":41,"stem":42,"icon":43},"Queue","/docs/operator-guide/queue","1.docs/2.operator-guide/3.queue","i-lucide-list-checks",{"title":45,"path":46,"stem":47,"icon":48},"Search","/docs/operator-guide/search","1.docs/2.operator-guide/4.search","i-lucide-search",{"title":50,"path":51,"stem":52,"icon":53},"Metrics","/docs/operator-guide/metrics","1.docs/2.operator-guide/5.metrics","i-lucide-chart-line",{"title":55,"path":56,"stem":57,"icon":58},"Promotion","/docs/operator-guide/promotion","1.docs/2.operator-guide/6.promotion","i-lucide-shield-check",false,{"title":61,"icon":62,"path":63,"stem":64,"children":65,"page":59},"Automation","i-lucide-bot","/docs/automation","1.docs/3.automation",[66,71,75],{"title":67,"path":68,"stem":69,"icon":70},"Automation levels","/docs/automation/automation-levels","1.docs/3.automation/1.automation-levels","i-lucide-gauge",{"title":72,"path":73,"stem":74,"icon":58},"Promotion & drift","/docs/automation/promotion","1.docs/3.automation/2.promotion",{"title":76,"path":77,"stem":78,"icon":79},"Approvals & spot-checks","/docs/automation/approvals","1.docs/3.automation/3.approvals","i-lucide-user-check",{"title":81,"icon":82,"path":83,"stem":84,"children":85,"page":59},"Reference","i-lucide-book-marked","/docs/reference","1.docs/4.reference",[86,91],{"title":87,"path":88,"stem":89,"icon":90},"Glossary","/docs/reference/glossary","1.docs/4.reference/1.glossary","i-lucide-book-open",{"title":92,"path":93,"stem":94,"icon":95},"Frequently asked questions","/docs/reference/faq","1.docs/4.reference/2.faq","i-lucide-circle-help",{"id":97,"title":98,"body":99,"description":466,"extension":467,"meta":468,"navigation":474,"path":475,"seo":476,"stem":477,"__hash__":478},"posts/3.blog/11.how-we-test-every-step-and-workflow.md","How every step and workflow is tested before it touches a customer",{"type":100,"value":101,"toc":456},"minimark",[102,107,121,141,145,152,188,203,207,226,256,267,271,277,299,310,314,317,379,383,402,432,442,449,453],[103,104,106],"h2",{"id":105},"the-premise-invariants-not-vibes","The premise: invariants, not vibes",[108,109,110,111,115,116,120],"p",{},"An LLM agent produces text, but a production workflow must produce ",[112,113,114],"em",{},"typed output"," that a downstream step, or a customer's decision, can trust. The approach here is not to review an agent's prose and call it good. The agent is run against fixed inputs, and ",[117,118,119],"strong",{},"invariants"," are checked on its structured, schema-validated output. If the output violates an invariant, the sample fails. If enough samples fail, the change does not ship.",[108,122,123,124,128,129,132,133,136,137,140],{},"The whole system sits in ",[125,126,127],"code",{},"core/evaluation",", and it is built around one function: ",[125,130,131],{},"run_scenario",". It seeds a real job through the real paths, real template keys, real ",[125,134,135],{},"JobTransition"," rows, a real claim, then runs the agent with stubbed tools, finishes the step, and applies the scenario's checks. Crucially, it ",[117,138,139],{},"never raises for agent failures",". A bad run becomes an outcome, not an exception.",[103,142,144],{"id":143},"what-gets-checked","What gets checked",[108,146,147,148,151],{},"The checks are deliberately narrow and encoded, not prompted. Three examples from ",[125,149,150],{},"core/evaluation/checks.py",".",[153,154,155,164,176],"ul",{},[156,157,158,163],"li",{},[117,159,160,151],{},[125,161,162],{},"assert_facts_grounded"," Every valued fact in the verified intel must cite a URL that was actually retrieved, and its quote must appear verbatim, whitespace- and case-normalised, in that source's text. A fact that points at something never fetched, or whose quote does not exist, fails.",[156,165,166,171,172,175],{},[117,167,168,151],{},[125,169,170],{},"assert_revenue_not_on_file"," The absence side of the same invariant. When no retrieved source mentions revenue, the revenue fact must be ",[125,173,174],{},"value=None",". A hallucinated figure fails outright.",[156,177,178,183,184,187],{},[117,179,180,151],{},[125,181,182],{},"assert_price_from_notes"," Any price in the offer draft must appear in the operator's own pricing notes. A ",[125,185,186],{},"£14,000"," that the operator never wrote down is a failure.",[108,189,190,191,194,195,198,199,202],{},"These are universal invariants attached to a template in ",[125,192,193],{},"CHECKS_BY_TEMPLATE",". ",[125,196,197],{},"synthesize"," always runs the grounding check. ",[125,200,201],{},"draft_offer"," always runs the price check. That mapping matters because it is exactly what a replay of a real job runs too.",[103,204,206],{"id":205},"fixed-scenarios-anonymised-fixtures","Fixed scenarios, anonymised fixtures",[108,208,209,210,213,214,217,218,221,222,225],{},"The fixed scenarios live in ",[125,211,212],{},"core/evaluation/scenarios.py"," as an ",[125,215,216],{},"AgentScenario"," registry. Each one pins the agent, the template, the requirements, the seeded inputs, and the checks, plus a ",[125,219,220],{},"max_failure_rate",", set to ",[125,223,224],{},"0.01"," for these examples.",[108,227,228,229,232,233,236,237,240,241,244,245,247,248,251,252,255],{},"The fixtures are committed and anonymised: a fictional ",[117,230,231],{},"\"Acme Widgets Ltd\""," with a Companies House filing and a crawled \"About\" page. ",[125,234,235],{},"synthesize.grounded"," gives the agent sources that ",[112,238,239],{},"do"," contain revenue, so the grounding check should pass. ",[125,242,243],{},"synthesize.missing_revenue"," gives it a filing with \"No accounts filed\" and a page that says \"contact us for a quote\", so any revenue figure is a hallucination and ",[125,246,170],{}," must fail it. ",[125,249,250],{},"synthesize.after_rejection"," re-runs the same scenario with feedback attached, \"the revenue figure was wrong last time\", to prove a rejection is actually learned from. ",[125,253,254],{},"draft_offer.price_from_notes"," tests that a price is lifted from pricing notes, not invented.",[108,257,258,259,262,263,266],{},"Real-world fixtures are a different beast and live in ",[125,260,261],{},"EVAL_FIXTURE_DIR",", deliberately ",[117,264,265],{},"never committed"," to version control. Anonymised data is checked in so anyone can reproduce a failure. Real customer data stays out of the repository.",[103,268,270],{"id":269},"the-replay-corpus-tested-against-production-history","The replay corpus: tested against production history",[108,272,273,274],{},"Synthetic cases are necessary but not sufficient. The corpus answers a sharper question: ",[112,275,276],{},"would the agent have passed on the jobs a human already decided?",[108,278,279,282,283,286,287,290,291,294,295,298],{},[125,280,281],{},"scenario_from_job"," rebuilds exactly what a real job's ",[117,284,285],{},"first attempt"," saw, selected by time, not the run's current state, which may have drifted since. It reconstructs the requirements and upstream outputs ",[112,288,289],{},"as of the first claim",", and attaches the universal checks for that template. ",[125,292,293],{},"replay_corpus"," gathers live jobs with at least one approval decision, newest first, using the ",[112,296,297],{},"first"," decision because later attempts were shaped by operator feedback. A case whose human verdict was \"rejected\" carries its rejection reason as the feedback on replay. \"Approved\" cases carry none.",[108,300,301,302,305,306,309],{},"This is the important part. The agent is judged against what production actually saw, not a curated example. Every corpus sample is stamped with ",[125,303,304],{},"source_job_id"," and ",[125,307,308],{},"case_verdict"," so the audit trail points back at a specific real decision.",[103,311,313],{"id":312},"a-testing-ladder-of-four-commands","A testing ladder of four commands",[108,315,316],{},"The management commands escalate from \"does one step work\" to \"does the whole workflow survive a batch\".",[153,318,319,343,363,371],{},[156,320,321,326,327,330,331,334,335,338,339,342],{},[117,322,323],{},[125,324,325],{},"wf_step"," runs one step once, verbosely. ",[125,328,329],{},"--scenario"," picks a registered scenario. ",[125,332,333],{},"--job"," replays a real job id. ",[125,336,337],{},"--show-prompt"," dumps every prompt and response. With ",[125,340,341],{},"--interactive"," you can simulate rejections in a loop, up to three, feeding the rejection reason back as feedback each round.",[156,344,345,350,351,354,355,358,359,362],{},[117,346,347],{},[125,348,349],{},"run_agent_batch"," runs a scenario or a replay corpus ",[117,352,353],{},"N times"," and gates on the failure rate. It writes an ",[125,356,357],{},"EvalSample"," per run, then reads the gate ",[112,360,361],{},"from the database",", so samples written by Celery workers count too.",[156,364,365,370],{},[117,366,367],{},[125,368,369],{},"wf_batch"," drives the entire workflow N times from a fixture, acting as the operator through the real services, approving and rejecting like a human would.",[156,372,373,378],{},[117,374,375],{},[125,376,377],{},"wf_run"," is the interactive operator REPL: a full run driven by hand, with every approval, rejection, and submission attributed to a real operator user.",[103,380,382],{"id":381},"the-gate-a-rate-not-a-feeling","The gate: a rate, not a feeling",[108,384,385,386,389,390,393,394,397,398,401],{},"Pass/fail is computed from a failure rate against a threshold, and the arithmetic is deliberately careful. In ",[125,387,388],{},"BatchSummary",", transient samples are ",[117,391,392],{},"excluded"," from the rate. A rate-limited provider says nothing about your agent, so ",[125,395,396],{},"judged = samples - transient",", and the rate is ",[125,399,400],{},"(check_failed + errors) / judged",". Failures caused by the agent count against you. The weather does not.",[108,403,404,405,408,409,412,413,416,417,419,420,423,424,427,428,431],{},"But the exclusion has a limit. If more than ",[125,406,407],{},"--max-transient"," (default ",[125,410,411],{},"0.2",") of samples are transient, the batch is ",[117,414,415],{},"inconclusive",". You cannot judge an agent on a provider outage, so ",[125,418,349],{}," exits ",[125,421,422],{},"2"," instead of pretending to pass or fail. A rate above the threshold exits ",[125,425,426],{},"1",", so CI can gate a merge directly. The batch writes an ",[125,429,430],{},"EvalRun"," with every sample saved for audit.",[433,434,440],"pre",{"className":435,"code":437,"language":438,"meta":439},[436],"language-text","samples:        100\npassed:         98\ncheck_failed:   1\ntransient:      1 (excluded from the rate)\nfailure rate:   1.0% (threshold 1.0%)\ngate:           PASS\n","text","",[125,441,437],{"__ignoreMap":439},[108,443,444,445,448],{},"For corpus runs, rejected cases are excluded from the gate. They have no \"right answer\" to check. Only ",[125,446,447],{},"case_verdict=\"approved\""," samples are judged.",[103,450,452],{"id":451},"what-this-buys-you","What this buys you",[108,454,455],{},"A customer never sees the first version of a step. They see the version that passed fixed invariants, that replayed hundreds of real historical decisions without regressing, and that a machine, not a reviewer's confidence, declared fit. That is the difference between shipping an agent and shipping a governed workflow. Every step is proven against a threshold before it ever touches a customer.",{"title":439,"searchDepth":457,"depth":457,"links":458},2,[459,460,461,462,463,464,465],{"id":105,"depth":457,"text":106},{"id":143,"depth":457,"text":144},{"id":205,"depth":457,"text":206},{"id":269,"depth":457,"text":270},{"id":312,"depth":457,"text":313},{"id":381,"depth":457,"text":382},{"id":451,"depth":457,"text":452},"The scenario harness and management commands that gate every agent workflow on measured failure rates.","md",{"image":469,"date":471,"badge":472},{"src":470},"https://images.unsplash.com/photo-1461749280684-dccba630e2f6?auto=format&fit=crop&w=1200&q=80","2026-09-30",{"label":473},"Engineering",true,"/blog/how-we-test-every-step-and-workflow",{"title":98,"description":466},"3.blog/11.how-we-test-every-step-and-workflow","PO3xcVA2UV2Ne-_ygCylypxWbK9d4eqBY94z0PwyI2I",[480,485],{"title":481,"path":482,"stem":483,"description":484,"children":-1},"You can't ask an expert to write down what they know. Capture what they do.","/blog/capture-what-experts-do","3.blog/10.capture-what-experts-do","Capturing expert judgement in a governed workflow, not a written manual, and turning craft into repeatable systems.",{"title":486,"path":487,"stem":488,"description":489,"children":-1},"AI can do the legwork. It can't do your judgment.","/blog/ai-cant-do-your-judgment","3.blog/7.ai-cant-do-your-judgment","Gathering and summarising information is now a commodity. The judgement layer is the real moat.",1793716435453]