Verify Before Done: The Gate That Fixes 90% of AI Agent Failures

I keep a file of every way an AI agent has burned me, each entry scored by Effect (how much it hurt) × Improvement (how much fixing it helps). It's currently past fifty entries.

One entry scores a perfect 100: verify against ground truth before saying "done" — run the actual thing, don't infer. Nothing else comes close. Here's why it dominates, and the gate I now run.

The failure that started the file

I asked whether a login fix was live. The agent told me it wasn't deployed and that the code worked. Both statements were wrong, and it had checked neither. It had read a line in a project config file saying "GitHub pushes do not auto-deploy", believed it over reality, and reported from that belief. The repo was Git-connected and had already auto-deployed twice.

Then, when the login failed on my end, it suggested I might have tested it wrong.

Three distinct failures — asserting without checking, trusting a stale doc over the live system, and blaming the user — and all three collapse into one root cause: narration substituting for observation.

Why models do this (it isn't laziness)

"I've completed the fix" is an extremely probable next sentence given a context window full of edits. The model isn't lying in any deliberate sense; it's producing the most likely continuation of a transcript that looks like successful work. This gets called execution hallucination, and it's structural, not a bug you can prompt away.

Worse, the training process actively rewards it. RLHF favours fluent, assertive completions over hedged ones, which produces a model that sounds certain past the edge of what it actually knows — the "polite liar" problem. Researchers have also found that models over-trust their own outputs, so self-critique without external grounding is shallow.

Which kills the obvious fix. "Double-check your work" doesn't work, because the thing doing the checking is the thing that's wrong.

The money pattern: a gate the model can't talk its way through

The guardrail is to never allow the AI to self-report task completion without independent system verification. That means the check must produce evidence that confident prose cannot fake.

curl -s -o /dev/null -w '%{http_code}' https://site.example/login# proves the server answered. proves nothing about the flow.curl -s -X POST https://site.example/api/auth \  -d '{"email":"real@user.test","password":"..."}' \  | jq -e '.access_token' && echo "AUTH OK"# proves the flow works, on the env the user is actually oncurl -s https://site.example/_astro/index.js | grep -c 'resolveAlias'# proves the DEPLOYED bundle contains the change

Three properties make a check real:

• It observes the system, not the intention. A build passing says the code compiles. It says nothing about whether the feature works.
• It runs on the environment the user is actually on. Half of my original confusion was which environment was being tested. "Works on my machine" is not a result.
• It produces a receipt. If the other party can't independently confirm the action happened, telling them it happened is a statement of trust, not evidence.

The four checks I actually run

Gawande's rule for checklists is to list only the killer items — five to nine critical steps, not a comprehensive task dump. The WHO surgical checklist cut deaths 47% precisely because it stayed short enough to actually get used. Mine is four:

1. Did I run the thing I changed, end to end?

   Through it, not around it. Testing the neighbouring path is how you certify a broken one.

2. Did I look at the live environment's real state?

   Deploy log, git remote, the live bundle. Not what a config file claims the environment does.

3. Did I read the actual output?

   Read it — don't infer it from the fact that you typed the command.

4. Can I paste the receipt?

   If you can't show it, you haven't verified it. This is the question that catches the other three.

Four questions. If any answer is no, the honest status is "I haven't checked yet" — which is a complete, professional sentence, and infinitely more useful than a confident wrong answer.

The smoke test, written out in full

"Automate the verification" is easy advice and vague. Here is the actual script. It's the one thing that turned verify-before-done from a rule I kept breaking into a rule I can't break.

#!/usr/bin/env bash# Usage: ./scripts/smoke.sh https://your-site.comset -euo pipefailBASE="${1:?usage: smoke.sh }"FAILED=0pass() { printf '  \033[32mPASS\033[0m  %s\n' "$1"; }fail() { printf '  \033[31mFAIL\033[0m  %s\n' "$1"; FAILED=1; }echo "smoke: $BASE"# 1. is it alive, and is it actually OUR build?code=$(curl -fsS -o /tmp/home.html -w '%{http_code}' "$BASE/" || echo 000)[ "$code" = 200 ] && pass "home 200" || fail "home returned $code"# 2. did the deployed bundle receive the change? (this is the one people skip)1if grep -rqs 'resolveAlias' /tmp/home.html || \   curl -fsS "$BASE/_astro/" 2>/dev/null | grep -qs 'resolveAlias'; then  pass "new code present in deployed output"else  fail "deployed output does NOT contain the change — you are testing a stale build"2fi# 3. exercise the critical flow END TO END, not around it3token=$(curl -fsS -X POST "$BASE/api/auth" \  -H 'content-type: application/json' \  -d "{\"email\":\"$SMOKE_USER\",\"password\":\"$SMOKE_PASS\"}" \  | jq -r '.access_token // empty')[ -n "$token" ] && pass "auth returns a token" || fail "auth did not return a token"# 4. authenticated read — proves the token is real, not just well-formedme=$(curl -fsS "$BASE/api/me" -H "authorization: Bearer $token" | jq -r '.email // empty')4[ "$me" = "$SMOKE_USER" ] && pass "authenticated /api/me echoes the user" \                          || fail "expected $SMOKE_USER, got '${me:-nothing}'"# 5. no server-rendered error text leaked into the pageif grep -Eqi 'internal server error|undefined is not|cannot read' /tmp/home.html; then  fail "error text found in rendered HTML"else  pass "no error text in HTML"fiexit $FAILED

Check 2 is the one that would have saved me the argument that started this whole file. "Is my change actually in the thing that's deployed" is a different question from "is the site up", and I had been treating them as the same question.

The four states you can be in

Most bad status reports come from collapsing these four into the word "done". They are not the same, and only one of them is done.

The reason this table is useful is that each row is a genuinely acceptable thing to say. "It builds, I haven't run it yet" is professional. It's only dishonest when it gets reported as the bottom row.

Make the gate impossible to skip

A script only helps if it runs. So take the decision away — bind it to the deploy and to a pre-claim hook.

# package.json
{
  "scripts": {
    "build": "astro build",
    "smoke": "./scripts/smoke.sh \"${SMOKE_BASE:-http://localhost:4321}\"",
    "deploy": "npm run build && netlify deploy --prod && npm run smoke"
  }
}

Chaining smoke onto deploy with && means an unverified deploy is a failed command, not a judgement call. And the agent-side version, so "done" can't be uttered without evidence:

#!/usr/bin/env bash
# .claude/hooks/pre-claim-done.sh — PreToolUse guard
# Blocks a completion claim unless a smoke run passed in the last 10 minutes.
STAMP=.claude/.last-smoke-pass

if grep -Eqi "(it'?s (now )?(live|working|deployed)|all set|✅ done|task complete)" <<<"$CLAUDE_MESSAGE"; then
  if [ ! -f "$STAMP" ] || [ $(( $(date +%s) - $(stat -f %m "$STAMP") )) -gt 600 ]; then
    echo "BLOCKED: no passing smoke run in the last 10 min. Run: npm run smoke" >&2
    exit 2
  fi
fi
exit 0

Is that heavy-handed? Yes. It also permanently ended a category of failure that three months of politely worded instructions did not, and the cost is one command I was supposed to be running anyway.

The same gate belongs inside any unattended loop. Each unit of an overnight run validates its own artefact before the commit — because at 3am there is nobody to catch a confident, empty output.

Test the disconfirming case

The subtle version of this failure is only looking for evidence that you succeeded. Textbook confirmation bias: people test the cases that confirm current beliefs rather than the ones that could falsify them.

So invert it. Before declaring done, list the ways it could still be broken, then actively try to break your own claim. For a login fix that means attempting the login and expecting failure. If you go in wanting the green tick, you'll find a way to read one.

The stronger version is to use a separate critic — a fresh sub-agent, or a human, whose only job is to refute the claim. Marking your own homework is exactly what fails here.

The catch: verification has to be cheap or it gets skipped

Here's what I got wrong for months. I knew the rule and still skipped it, because each check was a small manual hassle and I was moving fast. If a check costs a decision, it gets cut under pressure — which is precisely when you need it.

So automate it into one command. A script that hits the live URL, exercises the flow, greps the deployed bundle and prints pass/fail turns "verify before done" from a discipline problem into a keystroke. Better still, make it a deterministic hook that runs whether or not anyone remembers.

Prompts are probabilistic. Hooks are not. Anything I truly cannot afford to have go wrong belongs in the harness, not in a sentence I'm hoping the model honours on turn forty.

The verdict

You cannot prompt your way out of execution hallucination, because the model that would follow the prompt is the model that's confidently wrong. What works is structural: a gate between "I think I did it" and "I'm telling you I did it", producing evidence the model can't generate from vibes.

Build the gate once, script it, and hook it. Then "done" starts meaning something — and that single word doing real work is worth more than any model upgrade this year.

"Build passes" is not "it works".

Want verification gates wired into your AI-assisted build pipeline? Get in touch, or browse the work first.