WAYS OF WORKING · DEMO 07 NINO CHAVEZ
"Verified" is a sentence, not a fact. The most expensive failures in agent-assisted work share one shape: an artifact claimed something was checked, a charitable reader believed the sentence, and nobody re-derived the claim from source.
The claim
"Migration applied and verified — ACL diff confirms every revocation landed as specified."
→ or scroll to advance · ← to go back
02 THE FAILURE MODE
"An artifact can assert that something 'was verified against canonical source X.' Reading that assertion and treating it as evidence is a circular audit — you are verifying the artifact's verification, not verifying the claim itself. This fails when the artifact's self-check was wrong, incomplete, or misread." audit-discipline.md — a standing methodology rule, promoted after this failure shape recurred
The rule applies to every artifact that describes itself: code comments, a prior session's summary, another agent's report, a document's own "last verified" line. LLMs raise the stakes — "verified" is a token pattern an agent can produce without the act ever happening — and a reviewer inclined to be charitable will accept it. The fix is mechanical: grep the file, run the command, query the system. Checks beat assertions about having checked.
03 CASE STUDY
A security cleanup: revoke public execute rights on privileged database functions the linter flagged. The migration ran, the ACL diff came back exactly as specified — and production broke twice.
False assumption 1
"Server jobs run with server privileges." They didn't — the cron routes and billing checks ran through the request-scoped client, exactly the roles the revocation stripped. The organizer dashboard and a daily email job died quietly.
False assumption 2
"The ACL diff verifies the change." It verifies the migration did what it said — not that the application still works. Two different claims; only one was checked.
The durable rule
A privileged function is safe to revoke only if unreachable through every path — direct calls, policy predicates, view definitions, other functions' bodies — and the final check is runtime, not inventory.
The runtime is the one witness that can't be quoted out of context.
04 THE ONION
The remediation that followed is a tour of self-descriptions diverging from reality — every layer caught by one more mechanical check:
The grep said
"This function is called via RPC from the app — keep it granted."
The decision doc said
"Seven functions classified: revoke anonymous access, keep authenticated."
The new code's own comment said
"Authenticated callers only."
The closing check was live: anonymous requests fired against all six functions, each answering permission denied — with untouched controls confirmed unaffected. Inventory opens the investigation; runtime closes it.
05 RECURSION
The research agents that scoped this demo series produced findings. Before any number reached a page, it was re-derived from source — and three didn't survive contact:
The project docs said
"113 pkill invocations chasing stray servers."
The research agent said
"The mining tool was rebuilt as eleven throwaway scripts."
A quick query said
"Four agent-authored PRs in the production repo."
That's why every deck in this series ends with a colophon stating its numbers were re-derived on publication day. The discipline isn't decoration — it changed the content each time.
06 THE DISCIPLINE
Pull the canonical source
When a claim matters, go where the truth mechanically lives: grep the actual file, run the actual command, query the actual database. Never settle for the sentence about it — including sentences you wrote yourself, last session.
Compare independent derivations
The adversarial version, one operator's real practice: ask the same question to two independent agent sessions and compare answers. Divergence is a free audit — agreement earned the cheap way is worth little, but disagreement always points at something.
Tier your trust
Test-backed claims can be trusted outright — that's what enforcement buys (demo 06). Everything else carries provenance: verified-against-what, as-of-when. No provenance, no trust.
Agents make confident prose cheap — but they make re-derivation cheap too. One grep costs seconds. The economics favor checking; only habit favors believing.
07 HONEST LIMITS
Checking has a budget
Re-deriving every claim would consume the session. The discipline is scoped: claims that gate a decision, touch production, or will be repeated publicly get re-derived. The rest carries its provenance and waits.
Canonical sources lag too
The "canonical" doc can be a stale snapshot — that's exactly what the 113-vs-122 miss was. Prefer sources that regenerate (a live query, a fresh mine, the runtime) over sources that were once written.
The rule was learned by failing
This discipline wasn't designed in advance. It was promoted into standing methodology after the charitable-reviewer failure recurred across projects — including by the operator, including by the agent. Nobody is exempt, which is why it's written down.
08 YOUR VERSION OF THIS
If you never touch code
When an AI (or a report, or a vendor) says "verified," ask what the verification physically was. If the answer is a restatement of the claim, you have a sentence, not a check.
If you're technical
End every risky change at the runtime: a smoke-test against the live behavior, not a diff of the intended state. Diffs verify intentions; only execution verifies systems.
If you build systems
Make provenance a first-class field — verified-against-what, as-of-which-commit — and let the untiered claim be visibly untrusted. Where claims recur, promote the check into CI so belief stops being part of the loop.