[{"data":1,"prerenderedAt":1030},["ShallowReactive",2],{"navigation":3,"blog":96,"blog-posts":110},[4,23,60,80],{"title":5,"path":6,"stem":7,"children":8,"icon":22},"Getting Started","/docs/getting-started","1.docs/1.getting-started/1.index",[9,12,17],{"title":10,"path":6,"stem":7,"icon":11},"Introduction","i-lucide-house",{"title":13,"path":14,"stem":15,"icon":16},"Core concepts","/docs/getting-started/concepts","1.docs/1.getting-started/2.concepts","i-lucide-lightbulb",{"title":18,"path":19,"stem":20,"icon":21},"Your first run","/docs/getting-started/first-run","1.docs/1.getting-started/3.first-run","i-lucide-play","i-lucide-rocket",{"title":24,"icon":25,"path":26,"stem":27,"children":28,"page":59},"Operator Guide","i-lucide-layout-dashboard","/docs/operator-guide","1.docs/2.operator-guide",[29,34,39,44,49,54],{"title":30,"path":31,"stem":32,"icon":33},"Runs","/docs/operator-guide/runs","1.docs/2.operator-guide/1.runs","i-lucide-git-branch",{"title":35,"path":36,"stem":37,"icon":38},"Targets","/docs/operator-guide/targets","1.docs/2.operator-guide/2.targets","i-lucide-building-2",{"title":40,"path":41,"stem":42,"icon":43},"Queue","/docs/operator-guide/queue","1.docs/2.operator-guide/3.queue","i-lucide-list-checks",{"title":45,"path":46,"stem":47,"icon":48},"Search","/docs/operator-guide/search","1.docs/2.operator-guide/4.search","i-lucide-search",{"title":50,"path":51,"stem":52,"icon":53},"Metrics","/docs/operator-guide/metrics","1.docs/2.operator-guide/5.metrics","i-lucide-chart-line",{"title":55,"path":56,"stem":57,"icon":58},"Promotion","/docs/operator-guide/promotion","1.docs/2.operator-guide/6.promotion","i-lucide-shield-check",false,{"title":61,"icon":62,"path":63,"stem":64,"children":65,"page":59},"Automation","i-lucide-bot","/docs/automation","1.docs/3.automation",[66,71,75],{"title":67,"path":68,"stem":69,"icon":70},"Automation levels","/docs/automation/automation-levels","1.docs/3.automation/1.automation-levels","i-lucide-gauge",{"title":72,"path":73,"stem":74,"icon":58},"Promotion & drift","/docs/automation/promotion","1.docs/3.automation/2.promotion",{"title":76,"path":77,"stem":78,"icon":79},"Approvals & spot-checks","/docs/automation/approvals","1.docs/3.automation/3.approvals","i-lucide-user-check",{"title":81,"icon":82,"path":83,"stem":84,"children":85,"page":59},"Reference","i-lucide-book-marked","/docs/reference","1.docs/4.reference",[86,91],{"title":87,"path":88,"stem":89,"icon":90},"Glossary","/docs/reference/glossary","1.docs/4.reference/1.glossary","i-lucide-book-open",{"title":92,"path":93,"stem":94,"icon":95},"Frequently asked questions","/docs/reference/faq","1.docs/4.reference/2.faq","i-lucide-circle-help",{"id":97,"title":98,"body":99,"description":100,"extension":103,"meta":104,"navigation":105,"path":106,"seo":107,"stem":108,"__hash__":109},"blog/3.blog.yml","Blog",{"title":98,"description":100,"navigation":101},"Notes on automating expert work, measuring it honestly, and keeping a human where judgement matters.",{"icon":102},"i-lucide-newspaper","yml",{},{"icon":102},"/blog",{"title":98,"description":100},"3.blog","7xmmBP9X7bpUTRbjPj1doppz0TI1ujrl9Y10-4aLA6c",[111,223,599,719,871],{"id":112,"title":113,"body":114,"description":210,"extension":211,"meta":212,"navigation":218,"path":219,"seo":220,"stem":221,"__hash__":222},"posts/3.blog/10.capture-what-experts-do.md","You can't ask an expert to write down what they know. Capture what they do.",{"type":115,"value":116,"toc":200},"minimark",[117,122,126,129,132,136,139,142,145,149,152,155,158,162,165,168,171,175,178,181,187,191,194,197],[118,119,121],"h2",{"id":120},"the-written-answer-is-always-wrong","The written answer is always wrong",[123,124,125],"p",{},"Sit a founder down and ask them to write out how their business actually works, and you will get a document that is accurate and useless in equal measure. It will name the steps and miss the thing that makes the steps work. It will say \"review the schedule\" but not \"spot the row that looks fine and isn't\", because the person who wrote it has no idea they do that. It isn't a process to them. It's Tuesday.",[123,127,128],{},"That is the central problem with trying to industrialise a craft business. The value is locked inside people who cannot hand it over. Not because they're withholding it, but because they can't see it. Expertise feels like recognition. You just know a design code won't allow that, or that a creator is wrong for this brand. You can't ask someone to write down what they know, because what they know is mostly judgement, and judgement doesn't transcribe.",[123,130,131],{},"So stop asking. Watch what the expert does, and capture four things: their method, their judgement, their questions, and their red lines.",[118,133,135],{"id":134},"the-method-becomes-the-template","The method becomes the template",[123,137,138],{},"Start with the part the expert can describe: the sequence. Every craft business already has one, even if it's never been written down. An architect takes a brief, pulls house types from a library, checks them against the design code, generates options, and the client picks. A talent manager gathers a brief, finds the right creators, prices the deal, drafts the offer, and sends it.",[123,140,141],{},"That sequence becomes the workflow. In the engine it's a definition of steps, each with a name, a typed input, a typed output, and a number: how many minutes that step takes a human today. That number matters more than it sounds, because it is the baseline against which everything else is measured. You cannot claim a system saves time until you've frozen how long the work takes now. And the expert's own operating procedure, not a guess, is where that number comes from.",[123,143,144],{},"A real one: Pollard Thomas Edwards, an architecture practice, already automates its Schedule of Accommodation, a Revit-to-Excel table of every room and area in a building. The step they still do by hand is reconciliation: dropping the spreadsheet, checking it against the brief and the design code, and working out what doesn't add up. That manual judgement became a workflow of five steps: gather, read, validate, draft a \"what doesn't add up\" report, sign off. The legwork is the machine's. The sign-off is still theirs.",[118,146,148],{"id":147},"the-judgement-becomes-the-gate","The judgement becomes the gate",[123,150,151],{},"Now the part the expert can't describe. The sequence isn't the value. The judgement along the sequence is. So every point where a human would stop and make a call becomes an approval gate: the machine does the work, then stops and waits.",[123,153,154],{},"Notice what this does to the \"will it replace me\" fear. It inverts it. The expert doesn't hand over their judgement. They get promoted to the person who reviews it. A step only sheds its gate once it has earned it, by passing a scenario batch of a hundred cases and a replay corpus of thirty-plus approved decisions, then holding a spot-check floor. The gate is earned off, never switched off.",[123,156,157],{},"In practice this is where a professional-service business finally relaxes. The compliance check on a building runs through the regulations, flags the non-compliant clauses, and cites the exact text each finding came from. But the Principal Designer still signs off. In architecture that sign-off is legally load-bearing, post-Grenfell. \"The sign-off stays human\" isn't a limitation you apologise for. It's the reason a partner trusts the system at all.",[118,159,161],{"id":160},"the-questions-become-the-schema","The questions become the schema",[123,163,164],{},"Here's the part nobody writes down because it feels too obvious: the questions the expert always asks before they start. Which design code applies? What's the budget band? Which regulations? What's the goal of the offer?",[123,166,167],{},"Those questions are the expert's judgement made visible. In the engine they become a typed schema, the requirements the operator must supply before a run starts, not a free-text brief for the model to interpret. In the architecture example: client choices, the design code URL, the regulations to parse. In the talent example: the target, the goal, the pricing notes.",[123,169,170],{},"This is load-bearing for a simple reason. If you capture only the answers, the answers go stale. The expert who \"just knows\" what to ask is the one whose departure should frighten you most. Turning their questions into a schema is how you find out what they actually look at, because a requirement you can't turn into a field is a judgement you haven't understood yet.",[118,172,174],{"id":173},"what-theyd-never-let-go-wrong-becomes-the-test-suite","What they'd never let go wrong becomes the test suite",[123,176,177],{},"The last thing to capture is negative knowledge: the things the expert would never let out of the door. Not \"do a good job\". The specific failures. A schedule that matches the brief perfectly must not produce invented discrepancies. A compliant design must not be flagged as non-compliant. A clean spreadsheet must not crash because a cell is empty.",[123,179,180],{},"Each of those becomes a scenario: an input with a known-good outcome, run against the step, with a failure threshold. The expert doesn't write down their standards. They write down what wrong looks like, and the system refuses to ship output that trips it. This is where craft gets genuinely preserved, because it's where the expert's instinct, \"that feels off\", becomes a checkable assertion.",[182,183,184],"blockquote",{},[123,185,186],{},"You can't ask an expert to write down what they know. You can ask them what they'd never let go wrong, and watch what they do. And that's enough.",[118,188,190],{"id":189},"the-expert-doesnt-disappear-they-become-the-standard","The expert doesn't disappear. They become the standard",[123,192,193],{},"This isn't a way to get rid of expensive people. It's a way to stop being held hostage by them, including, if you're honest, the version of you that is the expert.",[123,195,196],{},"The founder reading this is usually that expert. You built the business on judgement you can't articulate, which is why it doesn't scale and why you can't step back. The point of the whole exercise is to turn what you do into something that runs when you're not in the room, with your judgement still holding the gates.",[123,198,199],{},"The whole point: you don't replace the expert, you industrialise them. The method becomes the templates, the judgement becomes the gates, the questions become the schema, and the red lines become the tests. Everything else, the drafting, the checking, the reconciliation, was never the craft. It was the tax on it.",{"title":201,"searchDepth":202,"depth":202,"links":203},"",2,[204,205,206,207,208,209],{"id":120,"depth":202,"text":121},{"id":134,"depth":202,"text":135},{"id":147,"depth":202,"text":148},{"id":160,"depth":202,"text":161},{"id":173,"depth":202,"text":174},{"id":189,"depth":202,"text":190},"Capturing expert judgement in a governed workflow, not a written manual, and turning craft into repeatable systems.","md",{"image":213,"date":215,"badge":216},{"src":214},"https://images.unsplash.com/photo-1522071820081-009f0129c71c?auto=format&fit=crop&w=1200&q=80","2026-10-07",{"label":217},"Craft",true,"/blog/capture-what-experts-do",{"title":113,"description":210},"3.blog/10.capture-what-experts-do","cO7telTGVXW3rgE8gB5ihFfvunmtqvLnMQ_idjjeHhw",{"id":224,"title":225,"body":226,"description":588,"extension":211,"meta":589,"navigation":218,"path":595,"seo":596,"stem":597,"__hash__":598},"posts/3.blog/11.how-we-test-every-step-and-workflow.md","How every step and workflow is tested before it touches a customer",{"type":115,"value":227,"toc":579},[228,232,245,265,269,276,312,327,331,350,380,391,395,401,423,434,438,441,503,507,526,556,565,572,576],[118,229,231],{"id":230},"the-premise-invariants-not-vibes","The premise: invariants, not vibes",[123,233,234,235,239,240,244],{},"An LLM agent produces text, but a production workflow must produce ",[236,237,238],"em",{},"typed output"," that a downstream step, or a customer's decision, can trust. The approach here is not to review an agent's prose and call it good. The agent is run against fixed inputs, and ",[241,242,243],"strong",{},"invariants"," are checked on its structured, schema-validated output. If the output violates an invariant, the sample fails. If enough samples fail, the change does not ship.",[123,246,247,248,252,253,256,257,260,261,264],{},"The whole system sits in ",[249,250,251],"code",{},"core/evaluation",", and it is built around one function: ",[249,254,255],{},"run_scenario",". It seeds a real job through the real paths, real template keys, real ",[249,258,259],{},"JobTransition"," rows, a real claim, then runs the agent with stubbed tools, finishes the step, and applies the scenario's checks. Crucially, it ",[241,262,263],{},"never raises for agent failures",". A bad run becomes an outcome, not an exception.",[118,266,268],{"id":267},"what-gets-checked","What gets checked",[123,270,271,272,275],{},"The checks are deliberately narrow and encoded, not prompted. Three examples from ",[249,273,274],{},"core/evaluation/checks.py",".",[277,278,279,288,300],"ul",{},[280,281,282,287],"li",{},[241,283,284,275],{},[249,285,286],{},"assert_facts_grounded"," Every valued fact in the verified intel must cite a URL that was actually retrieved, and its quote must appear verbatim, whitespace- and case-normalised, in that source's text. A fact that points at something never fetched, or whose quote does not exist, fails.",[280,289,290,295,296,299],{},[241,291,292,275],{},[249,293,294],{},"assert_revenue_not_on_file"," The absence side of the same invariant. When no retrieved source mentions revenue, the revenue fact must be ",[249,297,298],{},"value=None",". A hallucinated figure fails outright.",[280,301,302,307,308,311],{},[241,303,304,275],{},[249,305,306],{},"assert_price_from_notes"," Any price in the offer draft must appear in the operator's own pricing notes. A ",[249,309,310],{},"£14,000"," that the operator never wrote down is a failure.",[123,313,314,315,318,319,322,323,326],{},"These are universal invariants attached to a template in ",[249,316,317],{},"CHECKS_BY_TEMPLATE",". ",[249,320,321],{},"synthesize"," always runs the grounding check. ",[249,324,325],{},"draft_offer"," always runs the price check. That mapping matters because it is exactly what a replay of a real job runs too.",[118,328,330],{"id":329},"fixed-scenarios-anonymised-fixtures","Fixed scenarios, anonymised fixtures",[123,332,333,334,337,338,341,342,345,346,349],{},"The fixed scenarios live in ",[249,335,336],{},"core/evaluation/scenarios.py"," as an ",[249,339,340],{},"AgentScenario"," registry. Each one pins the agent, the template, the requirements, the seeded inputs, and the checks, plus a ",[249,343,344],{},"max_failure_rate",", set to ",[249,347,348],{},"0.01"," for these examples.",[123,351,352,353,356,357,360,361,364,365,368,369,371,372,375,376,379],{},"The fixtures are committed and anonymised: a fictional ",[241,354,355],{},"\"Acme Widgets Ltd\""," with a Companies House filing and a crawled \"About\" page. ",[249,358,359],{},"synthesize.grounded"," gives the agent sources that ",[236,362,363],{},"do"," contain revenue, so the grounding check should pass. ",[249,366,367],{},"synthesize.missing_revenue"," gives it a filing with \"No accounts filed\" and a page that says \"contact us for a quote\", so any revenue figure is a hallucination and ",[249,370,294],{}," must fail it. ",[249,373,374],{},"synthesize.after_rejection"," re-runs the same scenario with feedback attached, \"the revenue figure was wrong last time\", to prove a rejection is actually learned from. ",[249,377,378],{},"draft_offer.price_from_notes"," tests that a price is lifted from pricing notes, not invented.",[123,381,382,383,386,387,390],{},"Real-world fixtures are a different beast and live in ",[249,384,385],{},"EVAL_FIXTURE_DIR",", deliberately ",[241,388,389],{},"never committed"," to version control. Anonymised data is checked in so anyone can reproduce a failure. Real customer data stays out of the repository.",[118,392,394],{"id":393},"the-replay-corpus-tested-against-production-history","The replay corpus: tested against production history",[123,396,397,398],{},"Synthetic cases are necessary but not sufficient. The corpus answers a sharper question: ",[236,399,400],{},"would the agent have passed on the jobs a human already decided?",[123,402,403,406,407,410,411,414,415,418,419,422],{},[249,404,405],{},"scenario_from_job"," rebuilds exactly what a real job's ",[241,408,409],{},"first attempt"," saw, selected by time, not the run's current state, which may have drifted since. It reconstructs the requirements and upstream outputs ",[236,412,413],{},"as of the first claim",", and attaches the universal checks for that template. ",[249,416,417],{},"replay_corpus"," gathers live jobs with at least one approval decision, newest first, using the ",[236,420,421],{},"first"," decision because later attempts were shaped by operator feedback. A case whose human verdict was \"rejected\" carries its rejection reason as the feedback on replay. \"Approved\" cases carry none.",[123,424,425,426,429,430,433],{},"This is the important part. The agent is judged against what production actually saw, not a curated example. Every corpus sample is stamped with ",[249,427,428],{},"source_job_id"," and ",[249,431,432],{},"case_verdict"," so the audit trail points back at a specific real decision.",[118,435,437],{"id":436},"a-testing-ladder-of-four-commands","A testing ladder of four commands",[123,439,440],{},"The management commands escalate from \"does one step work\" to \"does the whole workflow survive a batch\".",[277,442,443,467,487,495],{},[280,444,445,450,451,454,455,458,459,462,463,466],{},[241,446,447],{},[249,448,449],{},"wf_step"," runs one step once, verbosely. ",[249,452,453],{},"--scenario"," picks a registered scenario. ",[249,456,457],{},"--job"," replays a real job id. ",[249,460,461],{},"--show-prompt"," dumps every prompt and response. With ",[249,464,465],{},"--interactive"," you can simulate rejections in a loop, up to three, feeding the rejection reason back as feedback each round.",[280,468,469,474,475,478,479,482,483,486],{},[241,470,471],{},[249,472,473],{},"run_agent_batch"," runs a scenario or a replay corpus ",[241,476,477],{},"N times"," and gates on the failure rate. It writes an ",[249,480,481],{},"EvalSample"," per run, then reads the gate ",[236,484,485],{},"from the database",", so samples written by Celery workers count too.",[280,488,489,494],{},[241,490,491],{},[249,492,493],{},"wf_batch"," drives the entire workflow N times from a fixture, acting as the operator through the real services, approving and rejecting like a human would.",[280,496,497,502],{},[241,498,499],{},[249,500,501],{},"wf_run"," is the interactive operator REPL: a full run driven by hand, with every approval, rejection, and submission attributed to a real operator user.",[118,504,506],{"id":505},"the-gate-a-rate-not-a-feeling","The gate: a rate, not a feeling",[123,508,509,510,513,514,517,518,521,522,525],{},"Pass/fail is computed from a failure rate against a threshold, and the arithmetic is deliberately careful. In ",[249,511,512],{},"BatchSummary",", transient samples are ",[241,515,516],{},"excluded"," from the rate. A rate-limited provider says nothing about your agent, so ",[249,519,520],{},"judged = samples - transient",", and the rate is ",[249,523,524],{},"(check_failed + errors) / judged",". Failures caused by the agent count against you. The weather does not.",[123,527,528,529,532,533,536,537,540,541,543,544,547,548,551,552,555],{},"But the exclusion has a limit. If more than ",[249,530,531],{},"--max-transient"," (default ",[249,534,535],{},"0.2",") of samples are transient, the batch is ",[241,538,539],{},"inconclusive",". You cannot judge an agent on a provider outage, so ",[249,542,473],{}," exits ",[249,545,546],{},"2"," instead of pretending to pass or fail. A rate above the threshold exits ",[249,549,550],{},"1",", so CI can gate a merge directly. The batch writes an ",[249,553,554],{},"EvalRun"," with every sample saved for audit.",[557,558,563],"pre",{"className":559,"code":561,"language":562,"meta":201},[560],"language-text","samples:        100\npassed:         98\ncheck_failed:   1\ntransient:      1 (excluded from the rate)\nfailure rate:   1.0% (threshold 1.0%)\ngate:           PASS\n","text",[249,564,561],{"__ignoreMap":201},[123,566,567,568,571],{},"For corpus runs, rejected cases are excluded from the gate. They have no \"right answer\" to check. Only ",[249,569,570],{},"case_verdict=\"approved\""," samples are judged.",[118,573,575],{"id":574},"what-this-buys-you","What this buys you",[123,577,578],{},"A customer never sees the first version of a step. They see the version that passed fixed invariants, that replayed hundreds of real historical decisions without regressing, and that a machine, not a reviewer's confidence, declared fit. That is the difference between shipping an agent and shipping a governed workflow. Every step is proven against a threshold before it ever touches a customer.",{"title":201,"searchDepth":202,"depth":202,"links":580},[581,582,583,584,585,586,587],{"id":230,"depth":202,"text":231},{"id":267,"depth":202,"text":268},{"id":329,"depth":202,"text":330},{"id":393,"depth":202,"text":394},{"id":436,"depth":202,"text":437},{"id":505,"depth":202,"text":506},{"id":574,"depth":202,"text":575},"The scenario harness and management commands that gate every agent workflow on measured failure rates.",{"image":590,"date":592,"badge":593},{"src":591},"https://images.unsplash.com/photo-1461749280684-dccba630e2f6?auto=format&fit=crop&w=1200&q=80","2026-09-30",{"label":594},"Engineering","/blog/how-we-test-every-step-and-workflow",{"title":225,"description":588},"3.blog/11.how-we-test-every-step-and-workflow","PO3xcVA2UV2Ne-_ygCylypxWbK9d4eqBY94z0PwyI2I",{"id":600,"title":601,"body":602,"description":708,"extension":211,"meta":709,"navigation":218,"path":715,"seo":716,"stem":717,"__hash__":718},"posts/3.blog/7.ai-cant-do-your-judgment.md","AI can do the legwork. It can't do your judgment.",{"type":115,"value":603,"toc":702},[604,608,611,614,617,622,626,629,632,652,655,659,662,665,668,671,675,678,684,690,696,699],[118,605,607],{"id":606},"the-split-everyone-gets-wrong","The split everyone gets wrong",[123,609,610],{},"Most conversations about AI in professional services start in the wrong place. They start with the model. Which one, how fast, how clever. They should start with a question about where value actually lives in the business.",[123,612,613],{},"Here is the uncomfortable fact. The part of the work everyone is most excited to hand to a machine, the gathering, the reading, the summarising, is the part that has already collapsed to a commodity. Any decent model can pull a Companies House filing, crawl a design code, or draft a competitor landscape in minutes. That work was never the moat. It was the legwork.",[123,615,616],{},"The moat has always been the layer above it: the judgement. Which creator fits this brief, not the fact that there are thousands of them. Whether an application clears the bar, not whether the form is complete. Whether a build option is actually buildable, not whether it complies on paper. The part you cannot commodity is the part you have been told to automate first.",[182,618,619],{},[123,620,621],{},"The mistake is not overestimating what AI can do. It is underestimating what you cannot hand it.",[118,623,625],{"id":624},"the-ladder-that-makes-the-split-explicit","The ladder that makes the split explicit",[123,627,628],{},"Stop asking \"can AI do my job?\" Ask \"which steps can run without me, which need me in the room, and which are still entirely mine?\"",[123,630,631],{},"That is the whole of it, expressed as three levels.",[277,633,634,640,646],{},[280,635,636,639],{},[241,637,638],{},"L1, fully autonomous."," The agent gathers, parses, and produces, and it ships without a human in the loop. Reading a spreadsheet, crawling public regulations, pulling candidate house types from a library.",[280,641,642,645],{},[241,643,644],{},"L2, agent plus an approval gate."," The machine does the legwork and then stops, hands the result to a human, and waits. A shortlist of build options. A compliance finding. A recommendation that still needs a sign-off.",[280,647,648,651],{},[241,649,650],{},"L3, fully manual."," The human's own step, untouched. The decision itself.",[123,653,654],{},"The ladder matters because it lets you draw the line honestly. Almost no real workflow is one thing end to end. A compliance check for an architecture practice is mostly automatable: parse the regulations, check the submission against them, draft the report. But it ends with a Principal Designer signing off, and that step is L3 because the liability is human and always will be. The machine does everything around the decision. It never makes it.",[118,656,658],{"id":657},"the-gate-is-a-feature-not-a-concession","The gate is a feature, not a concession",[123,660,661],{},"This is where the reframe bites. In most AI product pitches, the human approval gate is presented as a stopgap, a temporary safety measure you will remove once the model gets good enough. That framing is backwards, and it quietly tells your client you do not understand their business.",[123,663,664],{},"A gate is not where the system fails to be autonomous. It is where the system records, on purpose, that judgement lives here and has not been automated yet.",[123,666,667],{},"That is not a theoretical point. When a luxury concierge business runs 66 agents across its travel and dining curation, the interesting number is not the agents. It is the 154 approval gates. Each one is a mark on the map saying \"this is where a curator's taste cannot be automated yet.\" The gates are not friction. They are the elicited judgement. They are what lets you add clients without adding risk, because you know exactly where a human is still accountable.",[123,669,670],{},"There is a cleaner way to say it. The gate is not a limitation you tolerate. It is the feature that makes the whole thing safe to scale. And in some businesses it is literally the product. A vetted, members-only club does not want a machine deciding who gets in. It wants the machine to prepare the application, surface the risk flags, log the reason, and then stop, so a human can say yes or no and have that decision logged, timed, and consistent. The gate does not get shorter. It gets smarter.",[118,672,674],{"id":673},"what-this-means-if-you-are-the-one-building-it","What this means if you are the one building it",[123,676,677],{},"If you are tired of hearing \"AI!\" and now asking \"but how do I trust it?\", there is a concrete answer. It has three parts, and they have to chain together.",[123,679,680,683],{},[241,681,682],{},"Governance."," The rules are built in, so the system cannot misbehave. The workflow is a typed definition, not a prompt. Every step names its inputs and outputs, and a step cannot be configured to require something nothing produces. Either it compiles or it does not.",[123,685,686,689],{},[241,687,688],{},"Evals."," The system is measured against known answers before it ships. Not a vibes check. Scenario suites with held-out examples, run in batches, with a failure rate attached.",[123,691,692,695],{},[241,693,694],{},"Provenance."," Every answer shows its evidence, so you can re-check it later. A compliance flag points at the exact clause it came from, not at the model's confidence.",[123,697,698],{},"None of this is a sales pitch. It is the honest minimum for a system you would let near a real decision. The trick is that the human gate is not the part you are trying to engineer away. It is the part you build the whole thing around. The moment you can trace every automated step back to a source, and every human step back to a named person, is the moment the machine stops being a gamble and starts being an instrument.",[123,700,701],{},"That is the whole reframe. Collecting and summarising is now a commodity, and it should be. Spend your machines there. The judgement layer is the moat, and it has always been yours. The work is not to automate it. The work is to build the system that lets you scale it without letting go of it.",{"title":201,"searchDepth":202,"depth":202,"links":703},[704,705,706,707],{"id":606,"depth":202,"text":607},{"id":624,"depth":202,"text":625},{"id":657,"depth":202,"text":658},{"id":673,"depth":202,"text":674},"Gathering and summarising information is now a commodity. The judgement layer is the real moat.",{"image":710,"date":712,"badge":713},{"src":711},"https://images.unsplash.com/photo-1528819622765-d6bcf132f793?auto=format&fit=crop&w=1200&q=80","2026-10-28",{"label":714},"Strategy","/blog/ai-cant-do-your-judgment",{"title":601,"description":708},"3.blog/7.ai-cant-do-your-judgment","iMfPlWKxK5OKsk6Hlr2pGpGTQzHQi1GzjWPbX0LrONM",{"id":720,"title":721,"body":722,"description":861,"extension":211,"meta":862,"navigation":218,"path":867,"seo":868,"stem":869,"__hash__":870},"posts/3.blog/8.minutes-displaced-quality-held.md","Minutes displaced, quality held: the only AI metric worth reporting.",{"type":115,"value":723,"toc":854},[724,728,731,737,740,744,747,754,757,768,775,779,786,789,792,797,800,804,807,810,820,823,827,830,844,851],[118,725,727],{"id":726},"the-metric-everyone-reports-is-the-one-nobody-should-believe","The metric everyone reports is the one nobody should believe",[123,729,730],{},"Somewhere a vendor is telling you their agent \"predicts demand within a few per cent,\" or \"achieves 95% accuracy on invoices.\" These numbers are almost always computed on the vendor's own test data, against no agreed baseline, over no stated window. They tell you the model is good at a quiz it wrote itself. They tell you nothing about whether your firm shipped faster, or whether the work that left the building was any good.",[123,732,733,734],{},"The claim that matters for a craft or professional-services business is narrower and harder to fake. ",[241,735,736],{},"How many minutes of expert time actually left the building, and did the quality hold after they left?",[123,738,739],{},"That is two numbers, not one. Autonomy share, and spot-check acceptance. Everything else is decoration.",[118,741,743],{"id":742},"autonomy-share-minutes-displaced-not-minutes-promised","Autonomy share: minutes displaced, not minutes promised",[123,745,746],{},"Autonomy share is simple arithmetic. Automated minutes divided by a frozen manual-time baseline, over runs closed in a window.",[123,748,749,750,753],{},"The word that does the work is ",[236,751,752],{},"frozen",". The baseline is captured when a service request closes, from the workflow's templates. Each step's execution minutes and orchestration minutes, stored as a snapshot at that moment. The point of freezing it is precisely that it must not drift upward as you optimise. If your baseline moved every time the agents got faster, autonomy share would quietly trend toward 100% while telling you nothing. A frozen baseline is the difference between a measurement and a moving target.",[123,755,756],{},"The honest part is in the credit rules. Not every minute counts as automated. The scorer walks each baseline slot and credits autonomy only where it actually happened.",[277,758,759,762,765],{},[280,760,761],{},"A step a human overrode earns no automation credit at all. A human did it, so the minutes go back to the baseline column.",[280,763,764],{},"A step completed at the highest automation level earns its execution minutes, and its orchestration minutes too.",[280,766,767],{},"Every rejection or failure a human had to clean up adds a charged attempt, because that is real expert time spent.",[123,769,770,771,774],{},"Automatic retries after a transient error are ",[236,772,773],{},"not"," charged. A provider rate limit cost no human time. The metric is trying, with unusual care, to measure displacement rather than activity. It wants to count the minutes your experts actually got back, not the minutes the machine was busy.",[118,776,778],{"id":777},"spot-check-acceptance-quality-held-checked-after-the-fact","Spot-check acceptance: quality held, checked after the fact",[123,780,781,782,785],{},"Displacement alone is a dangerous number, because it rewards a machine for doing lots of work badly. So the second number is spot-check acceptance. A human re-reviews a sample of outputs ",[236,783,784],{},"after"," they have shipped, and the share they accept is tracked.",[123,787,788],{},"The sampling is deliberately not random in a way anyone can game. When a fully-automated step completes, the system hashes the job's identifier and checks the result against a per-template sampling rate. Because the hash is reproducible, retrying the same job cannot dodge the review. And the review is non-blocking. The verdict is recorded, but it never changes the job or the run. The work has already left the building. The spot check exists to find out whether it should have.",[123,790,791],{},"Two details matter. First, a rejection requires a reason. The system refuses a bare \"no,\" because the rejection feeds back into future checks and the training corpus. Second, the two numbers are only meaningful read together. This is where the implementation is candid in a way almost no marketing page is.",[182,793,794],{},[123,795,796],{},"\"A rising share with falling spot-check acceptance means automation is getting ahead of quality.\"",[123,798,799],{},"That single sentence is the whole discipline. Displacement up, acceptance down: you are not saving time, you are exporting rework. The machine is signing off work your experts would have rejected.",[118,801,803],{"id":802},"demand-the-baseline-and-the-baseline-is-the-discovery-question","Demand the baseline, and the baseline is the discovery question",[123,805,806],{},"Here is the uncomfortable truth the metric's own documentation states plainly. The minutes are template estimates, not stopwatches. Operator time-logging was dropped as unreliable, so there is no reconciliation against actuals. The L2 credit is a judgement call, not a measurement. The metric knows what it is and says so.",[123,808,809],{},"That candour is exactly what you should demand from any vendor, and exactly what a vendor should demand from you. The reason most firms cannot tell you their autonomy share is not that the maths is hard. It is that they have never written down how long the manual process actually took. That missing number is the baseline.",[123,811,812,813,816,817],{},"So the first, and best, question to ask any AI vendor is not \"what is your accuracy?\" It is: ",[236,814,815],{},"walk me through your baseline."," How was the manual time measured? When was it frozen? What counts as automated, and what gets charged back when a human has to step in? And then: ",[236,818,819],{},"show me the spot-check acceptance next to the autonomy share, on the same chart, over the same window.",[123,821,822],{},"A vendor who can answer that already knows the number cannot be trusted on its own. A vendor who cannot has never actually measured the thing they are selling. That question, \"where is your baseline?\", is the discovery exercise that turns a demo into a paid pilot, because answering it requires looking at your actual process, your actual minutes, your actual rework.",[118,824,826],{"id":825},"the-only-honest-report","The only honest report",[123,828,829],{},"The metric worth reporting is not \"the model is X% accurate.\" It is two numbers read together.",[277,831,832,838],{},[280,833,834,837],{},[241,835,836],{},"Autonomy share."," The minutes of expert time displaced, against a baseline frozen before you started optimising.",[280,839,840,843],{},[241,841,842],{},"Spot-check acceptance."," The share of sampled, shipped outputs a human still agrees with.",[123,845,846,847,850],{},"Report them on the same chart, over the same window, with the baseline's limitations printed underneath. If autonomy share is rising and acceptance is holding, you have displaced real work. If the two diverge, you have found automation running ahead of quality, and you have found it ",[236,848,849],{},"before"," a client did.",[123,852,853],{},"That is the entire job of AI governance in a professional-services firm. Make the displacement real, and make the quality check late enough to be honest. Everything a model \"predicts within X%\" is a claim about the model. These two numbers are a claim about your business.",{"title":201,"searchDepth":202,"depth":202,"links":855},[856,857,858,859,860],{"id":726,"depth":202,"text":727},{"id":742,"depth":202,"text":743},{"id":777,"depth":202,"text":778},{"id":802,"depth":202,"text":803},{"id":825,"depth":202,"text":826},"Demand a frozen baseline and a post-hoc quality check before you trust any AI efficiency number.",{"image":863,"date":865,"badge":866},{"src":864},"https://images.unsplash.com/photo-1501139083538-0139583c060f?auto=format&fit=crop&w=1200&q=80","2026-10-21",{"label":50},"/blog/minutes-displaced-quality-held",{"title":721,"description":861},"3.blog/8.minutes-displaced-quality-held","2KO62pfQYt6SU-peeFrqMflDFx5gw-dA338BnNWUeyI",{"id":872,"title":873,"body":874,"description":1020,"extension":211,"meta":1021,"navigation":218,"path":1026,"seo":1027,"stem":1028,"__hash__":1029},"posts/3.blog/9.how-a-step-earns-autonomy.md","How a step earns the right to run unattended.",{"type":115,"value":875,"toc":1012},[876,880,883,890,893,899,903,906,919,925,928,932,949,958,962,965,980,985,988,992,999,1002,1006,1009],[118,877,879],{"id":878},"the-gate-is-a-lease-not-a-badge","The gate is a lease, not a badge",[123,881,882],{},"Every step in a workflow sits at one of three levels. L3 is fully manual: a person does the work. L2 is the interesting middle: an agent does the work, and a human must approve its output before the run advances. L1 is the prize. The agent runs, and the workflow moves on without anyone looking over its shoulder.",[123,884,885,886,889],{},"A step does not start at L1, and it does not get there by assertion. In the model it is a promotion: a documented, evidential decision that an L2 step may drop its approval gate. The object that records it is called ",[249,887,888],{},"PromotionEvidence",", and its docstring is unambiguous. \"Why an L2 step was allowed to drop its approval gate.\"",[123,891,892],{},"That promotion is never permanent. It is a lease, and it is revoked the moment the conditions it was granted under stop holding.",[123,894,895,896,898],{},"A note on what is live today: this post describes the model the platform is built around. At the time of writing promotion is not yet operator-facing — a step is promoted by setting its ",[249,897,888],{}," in the workflow definition (a configuration change), and nothing ships pre-promoted. The drift and fallback machinery below is implemented but only comes into effect once a step has been promoted.",[118,900,902],{"id":901},"three-bodies-of-evidence","Three bodies of evidence",[123,904,905],{},"Promotion requires three separate proofs, all collected on the exact prompt and model the agent is running at the time.",[277,907,908,913,916],{},[280,909,910,911,275],{},"A fixed-scenario batch of at least 100 runs, gated on a ",[249,912,344],{},[280,914,915],{},"A replay corpus of at least 30 approved real cases, rebuilt exactly as their first attempt saw them.",[280,917,918],{},"Live step acceptance at or above threshold across at least 30 real, human-decided decisions.",[123,920,921,922,924],{},"The fixed scenarios are committed and anonymised, a fictional \"Acme Widgets Ltd\", so they can run deterministically. Each carries a ",[249,923,344],{},": one per cent in the reference fixtures. If more than one run in a hundred fails an invariant such as \"facts are grounded in the source\" or \"the price is taken from the pricing notes, not invented\", the batch does not pass and the promotion does not happen.",[123,926,927],{},"The replay corpus is the stronger half. It does not use synthetic examples. It pulls real, human-decided jobs and rebuilds the exact scenario their first attempt saw, selected by time so later feedback cannot leak into the evaluation. Two details matter. First, only the human's first decision counts, because later attempts were shaped by feedback. Second, human-written outputs are excluded outright. An overridden job is not agent evidence. You cannot promote an agent on work a person finished by hand.",[118,929,931],{"id":930},"what-gets-frozen","What gets frozen",[123,933,934,935,937,938,941,942,944,945,948],{},"When the step promotes, ",[249,936,888],{}," freezes the world it was proven in. It stores the prompt hash, a content hash of the prompt template, the model the agent was configured to call, and ",[249,939,940],{},"served_model",", the model the provider was actually serving. It stores both evaluation run IDs, the shared ",[249,943,344],{},", and a ",[249,946,947],{},"min_spot_check_acceptance"," floor for the monitoring that follows.",[123,950,951,952,429,955,957],{},"The distinction between ",[249,953,954],{},"model",[249,956,940],{}," is worth noticing. One is what you asked for. The other is what the provider actually ran. Both are captured, because both can drift.",[118,959,961],{"id":960},"drift-is-what-revokes-it","Drift is what revokes it",[123,963,964],{},"A promoted step is designed to be checked on two fronts.",[123,966,967,968,971,972,975,976,979],{},"The first is a hard gate, evaluated at instantiation. ",[249,969,970],{},"promotion_problems()"," compares the agent's current prompt hash and its ",[249,973,974],{},"default_model()"," against what the step was promoted on. Any mismatch, a rewritten prompt, a moved model default, returns a problem, and ",[249,977,978],{},"effective_requires_approval()"," falls back to L2. Crucially, the step keeps running. It does not block the workflow. It simply requires a human to approve its output again, exactly as it did before promotion.",[182,981,982],{},[123,983,984],{},"\"A promotion that no longer holds falls back to L2: the step still runs, a human approves it.\"",[123,986,987],{},"The second front is a scheduled drift check. It watches three things. Whether the most common served model over the last 24 hours has moved from the one recorded at promotion. Whether a re-proof batch has run in the last seven days. And whether live spot-check acceptance has fallen below the floor. If any of those breaks, the step is flagged and re-proven against the same fixed scenarios and a fresh sample of approved corpus cases.",[118,989,991],{"id":990},"spot-checks-close-the-loop","Spot checks close the loop",[123,993,994,995,998],{},"Promotion does not mean the step stops being watched. It means watching gets cheaper. Every promoted step carries a ",[249,996,997],{},"spot_check_rate",": a share of its L1 outputs is routed to a human for after-the-fact review. The sampling is deliberately un-gameable. It hashes the job ID, so the decision to check is reproducible and cannot be gamed by retries.",[123,1000,1001],{},"Those spot-check decisions are non-blocking. They never change the job or the run. But they are not inert. A rejection requires a reason, and that reason feeds the replay corpus and new checks. The same human judgement that once approved every output now samples a fraction of it, over a 30-day window, and the accumulated acceptance rate is the number that holds the step to its floor.",[118,1003,1005],{"id":1004},"governance-not-vibes","Governance, not vibes",[123,1007,1008],{},"The whole thing is a closed loop. A step earns L1 with a hundred deterministic runs and thirty real cases. It keeps L1 only while the prompt, the model, and the served model stay what they were proven as, and while humans keep accepting its sampled output above an agreed floor. When any condition breaks, the gate is meant to come back — automatically, without a meeting, without anyone deciding the step \"feels off\".",[123,1010,1011],{},"That is the difference between wrapping an LLM and governing one. Most tools can call a model. The hard part is knowing when it is safe to stop watching, being able to prove it, and having the machinery to pull the approval back the moment that proof stops holding.",{"title":201,"searchDepth":202,"depth":202,"links":1013},[1014,1015,1016,1017,1018,1019],{"id":878,"depth":202,"text":879},{"id":901,"depth":202,"text":902},{"id":930,"depth":202,"text":931},{"id":960,"depth":202,"text":961},{"id":990,"depth":202,"text":991},{"id":1004,"depth":202,"text":1005},"How a workflow step earns promotion to unattended operation, and what drift is designed to pull back to human approval.",{"image":1022,"date":1024,"badge":1025},{"src":1023},"https://images.unsplash.com/photo-1454165804606-c3d57bc86b40?auto=format&fit=crop&w=1200&q=80","2026-10-14",{"label":594},"/blog/how-a-step-earns-autonomy",{"title":873,"description":1020},"3.blog/9.how-a-step-earns-autonomy","EOmj86DA-0qxFCgIRoeG51hhoMPsZdfKa6q74VaBJOQ",1793716435012]