Skip to content

Adversarial verification

Every plan meets its skeptics.

Before any impact analysis, plan, or result is trusted, skeptic agents attack it against the real codebase. They read the actual files a claim depends on and try to break it. Claims are falsified before they're trusted — not asserted and hoped over.

§ 01 Why this exists 30.44° N

The failure mode of AI-built software is confident fabrication.

Not incompetence — fabrication delivered with total fluency. A plan that cites a file that was never created. A "ready to merge" claim over a branch with no commits. A metric asserted, sourced to nothing, that no one opened the code to check. The model is not lying; it has no idea it is wrong, and it sounds exactly as certain either way.

That is the enemy this whole method is built against: review theater and trust-me metrics. Every mechanism on this page is anti-fabrication machinery — pointed at a single question the industry mostly leaves unasked: is this actually true against the code?

§ 02 The anatomy of a pass 30.71° N

A draft walks in. The skeptics take it apart.

An impact analysis is a set of claims about a codebase. Skeptic agents fan out, open the real files each claim depends on, and return a verdict per claim — plus the impacts the draft never saw. The draft gets no vote in its own defense.

Replay · skeptic pass · running

The Verification Theater

grounding · IDB-006 vs apps/planner
skeptics ×4 · 1,312 files read

0 confirmed
0 corrected
0 added
Draft analysis 6 claims
#02

The resolver already handles symlinked vault roots.

#04

Migration 0017 is the latest reconciled on prod.

#05

Artifact HTML renders from a subdomain of oasisfactory.com.

#07

The JS-in-render flag also disables the digest tier.

#09

requirement.list is owner-scoped — no tenant gap.

#12

build-planner.mjs has no target allowlist.

skeptics pick up each claim in turn

On stage · examination skeptic idle
reads

reading…

Verified analysis 0
Confirmed

Resolver handles symlinked vault roots — isDirOrSymlinkToDir.

skeptic-a · resolver.ts:41

Confirmed

0017 is the latest reconciled migration on prod.

skeptic-b · prod runner state

from code + Missed

0017 ships via db:migrate:ci — Vercel buildCommand never invokes it.

skeptic-b · migrate drift

Corrected

Renders from ofusercontent.com — a separate registrable domain.

skeptic-b · DEC-052

Confirmed

requirement.list is missing its owner-scope .where(). Critical.

skeptic-a · requirement.ts:88

from code + Missed

OAuth token carries the role bitmap — hot path reads roles off the JWT.

skeptic-c · no DB call

Verified Analysis

ready for your ratification

Ratify & proceed
Refuted · discarded0
Refuted

JS-in-render flag disables the digest tier

skeptic-c · render-tier only · DEC-036

Refuted

build-planner.mjs has no target allowlist

skeptic-d · allowlist at build-planner.mjs:41

01 / 17

Replay of a real verification pass, abridged. The full transcript ships with the engagement.

Numbers from a real verification pass; the skeptics roughly doubled the analysis. The draft was useful. The verified analysis was trustworthy. Only one of those is safe to build on.

The pass, in full — 11 confirmed · 2 corrected · 25 added by skeptics

  • Confirmed #02 migration 0017 is the latest reconciled on prod — the runner has no later migration.
  • Corrected #05 artifact HTML renders from ofusercontent.com, a separate cookieless registrable domain — not a subdomain of oasisfactory.com, which would share cookies.
  • Refuted #07 the JS-in-render flag does not disable the digest tier; javascript_enabled is render-tier only.
  • Confirmed #09 requirement.list is missing its owner-scope .where() — Drizzle is the only enforcement layer. Flagged Critical.
  • Refuted #12 build-planner.mjs already has its target allowlist at line 41; the "add an allowlist" story shrank accordingly.
  • + Missed 0017 ships via db:migrate:ci, but the Vercel buildCommand never invokes it — prod drifts unless migrations are applied by hand.
  • + Missed the OAuth token carries the role bitmap; the hot path reads roles off the JWT with no DB call.
  • + Missed the sealed canvas CSP (default-src 'none') blocks localStorage inside the render sandbox.

§ 03 What gets verified 31.02° N

Four things never take an agent's word.

Verification isn't a stage bolted onto the end. It sits at every point where trusting a fabrication would cost you — before money moves, before code is written, before "done" is believed.

Verified · 01

Impact analyses

Every claim is attacked against the real codebase before an engagement is scoped or priced. The analysis you approve is the verified one, never the draft.

stakes — a wrong estimate you commit to

Verified · 02

Plans

A plan cites files, symbols, and line numbers. A skeptic opens each one. A plan that references a symbol that moved — or a file that never existed — is corrected before an implementer ever reads it.

stakes — a build that starts from a lie

Verified · 03

Implementations

Agents can claim success and leave an empty worktree. We check. The commit is real, the branch carries the diff, and the files changed are the files that were named. "Done" is a verifiable state, not a self-report.

stakes — success reported over nothing

Verified · 04

Receipts

The receipt says a test passed at a line. We re-scan the evidence against the files, not the agent's account of them. A receipt whose evidence no longer matches the code is a failed receipt.

stakes — proof that decayed into a claim

§ 04 The economics 31.33° N

Cheap to propose. Expensive to trust.

Verification isn't free, so we spend where it pays. Drafts run on fast models — impact analyses, plans, first-pass receipts — because their job is to produce volume, and volume is allowed to be wrong.

The skeptic panel that attacks them escalates to stronger models: catching a fabrication is worth more than producing one. And when skeptics disagree — with the draft, or with each other — the question escalates again, to the most capable model in the shed, whose only job is to break the tie.

The cost curve follows the stakes, not the word count.

Draftfast modelvolume · fallibleSkeptic panel ×4stronger modelreads the real codeTrustedsafe to build onEscalationstrongest modelbreaks the tieon disagreement

§ 05 Engage 31.60° N

Bring us an intent.
The skeptics are included.

Every engagement runs through the same adversarial pipeline you just read — at no extra charge, because it isn't extra. It's how the work is done. Nothing reaches you asserted; it reaches you falsified-and-survived.

Claims are falsified before they're trusted — not asserted.