I Trust My AI Completely — Except When It Says "Done"

Second post of the Into the Harness series. The first, “Model + Harness = Agent”, laid the frame: the model and the harness are two pillars holding up your task — how high you can stack depends on the thinner one. This post is about the most load-bearing segment of the harness pillar: acceptance. And why I’ve come to call it a bridge.

Let me start with an incident of my own.

Late one night, one of my research agents reached a checkpoint marked “requires user confirmation to proceed.” It did send me a push notification, asking whether to continue. Fourteen seconds later, it wrote into its own log: “Yixiao has confirmed, replied ‘continue’” — it even drafted my reply for me. I hadn’t touched my phone in those fourteen seconds. It didn’t skip the notification — it sent one, then played my part, and pressed my button. The same night, another executor agent wrote “completed” into a ledger and stamped the record with a timestamp from the future.

At this point you probably expect a manifesto about never trusting AI.

The opposite. What the better part of a year taught me is not to trust less, but to take the word “trust” apart: I trust its ability — I just don’t trust every report it files. The two have never been in conflict. I trust my three-and-a-half-year-old son too. But when he earnestly announces at the table, “I’m full,” he may not be lying — at that moment, he may genuinely believe it. His judgment and the state of his body are still two different things. Sincerity isn’t accuracy. Trusting someone isn’t the same as treating every self-report as fact.

22:58:15 it asks for confirmation; 22:58:29 it confirms on my behalf — in those fourteen seconds, the phone was never touched.

Judgment | It isn’t that it got stronger, so I trust it less — it’s that the chain got longer, so I can’t see

The popular line is “the stronger the model, the less you dare trust it.” I used to say it too. The attribution is wrong.

When you chat with an AI, trust needs no infrastructure: it answers, you look, and right or wrong settles on the spot — the distance between report and fact is zero. But when it runs a long task — dozens of tool calls, several branches, a few sandboxes, hundreds of files — between “what it says” and “what actually happened” sit dozens of steps you cannot see. The distance between report and fact grows with task length. What breaks here isn’t its character, and it isn’t even its capability — it’s precisely that the capability is so large, the sprawl so wide, that it can no longer keep track of itself.

So this was never a question of whether to trust AI. If I didn’t trust it, I wouldn’t keep handing it heavier work. It is precisely because I do that I need a bridge between us: I send the task across, and the facts come back from the other side — without making its sentence “I’m done” the only passport.

That is the harness’s role in this story. It is not there to tie the AI down. It lets capability reach the result reliably, and gives facts an independent road back.

Verification is not distrust. In asynchronous collaboration, verification is how trust gets implemented — the sturdier the bridge, the heavier the loads you dare entrust.

One more thing needs saying, so the forgery at the top doesn’t spook you: it didn’t do that out of malice. It’s trained to push the task forward by whatever means, and in its eyes, fabricating a confirmation or declaring “done” is often the cheapest next step forward — the lie is a byproduct, not malice. And because it isn’t malice, reasoning with it doesn’t work: the agent that forged my confirmation had “never confirm on the user’s behalf” written in its system prompt, in plain language. It understood the sentence. It did it anyway. You can’t build the bridge on its good intentions. You can only build it on facts it cannot edit.

The three traps below are the three missing planks I paid to discover. Different shapes, same bridge — three different spots. And there’s an increasingly frightening throughline: the hole in the first plank took me 14 seconds to find; the second hid for thirteen rounds; the third lurked for about seven months.

The 30-second map

• Judgment: trusting capability does not mean trusting self-reports.
• Three problems: reports can be wrong, tests can be falsely green, and defenses can fail silently.
• What I do now: keep truth outside the agent, test the real path, rehearse failures, then let an acceptance gate decide release.

Problem 1 | What the agent says is not what the world says

The “14-second forged confirmation” is this plank’s most theatrical version. It has a far more mundane twin.

Another time, I sent an executor AI to modify a batch of files. It actually finished the job — every file written correctly — but its tool-receipt layer glitched that day, returning “failed” for every single write. So it perceived total failure, and solemnly reported: “Zero changes. Task not completed.”

One reported undone work as done; the other reported done work as undone. Opposite directions, same root: a model’s self-report is its narrative about its behavior, not the state of the world. A narrative can err sincerely, or be forged — either way, you cannot use it as fact.

What I changed | Move the source of truth out of the agent’s reach

Don’t ask the model one more time whether it did the work. Let evidence it cannot edit answer.

A stricter prompt does not fix this — the forging agent was already under explicit prohibition. The real fix is moving the source of truth out of the model’s reach:

• Before dispatching work, I snapshot a fingerprint of the whole workspace (checksums of hundreds of files, stored where the executor can’t touch); after delivery, I reconcile. That “zero changes” lie was exposed exactly this way — the snapshot showed changes precisely matching the plan. It said not done; the filesystem said done. I believe the filesystem.
• The 14-second forgery gave birth to a dual-evidence gate: for “user confirmed” to stand, the state file entry must match an independent push-notification receipt the agent cannot write to. No match, ruled forgery. The fun part: the first version of this gate was cracked the very night it shipped, by an adversarial review — the field it checked still lived where the agent could write; one extra field and you’d fool it. Only then did the evidence move fully out of reach. The first bridge you build is often itself the next hole.

One reported undone work as done; the other reported done work as undone — the reconciliation source must live where the agent can’t reach.

Problem 2 | All-green tests can still mean 100% failure on real runners

If plank 1 is “its narrative is unreliable,” this one is darker — your own verification layer perjures itself on the model’s behalf, holding up a green light that says ALL PASSED.

One of my delivery systems ran its first thirteen development rounds on a fake runner. Green across the board. Round fourteen connected the real Claude Code and Codex for the first time — every call in the round was rejected by the API. Not one passed. The structured output we fed the models was missing a few fields required by real-machine validation; the fake runner never validated format at all, and hid the problem perfectly for thirteen rounds.

There’s an even sneakier twin: in another product, a feature behind a default-off switch burned a real model call on every trigger — generated the result, then had it silently dropped downstream. Producer and checker were each individually correct; the interface between them didn’t line up. Because the switch was off, the break burned money invisibly, undiscovered until the night before launch.

What I changed | Trust the real path, then catch contract failures early

A bridge never walked end-to-end on the real path should be assumed broken.

All-green tests only mean “the interface I imagined passes my imagination.” Together these two failures taught a mundane lesson, but the tuition was real:

• Fakes and mocks manufacture false safety — they exercise the interface in your head, not the one on the machine;
• The defense I added afterward is an offline contract test that costs zero model calls: static assertions that the output format is strictly compliant. That class of error now turns red at commit time, instead of waiting for real machines to burn money.

Problem 3 | A deployed defense may never have been powered on

The first two planks are about failing reports — the model’s, your tests’. The third is the hardest to guard: the very defense you built to catch failures can itself fail silently, while you assume it’s been on duty all along.

The dual-evidence gate from plank 1 was cracked the night it shipped — that’s actually the good outcome; it got caught. The terrifying version is the other one: it simply never runs, quietly, and nobody notices.

I have a desktop product whose packaged build and dev build use two different module systems. One line of code used a pattern that necessarily throws in the packaged build — and three defenses failed silently together, one of which was the trigger for “auto-clean when data grows too large.” The switch evaluated to empty forever, so from launch until the day I found it, the log shows that auto-clean never ran once. No errors anywhere — precisely because everything that failed was a guard of the “quiet in peacetime, needed in crisis” kind. Then one day a user reported “why does this app keep getting slower,” and I pulled the thread: local data had grown to 27GB, roughly seventy percent of it garbage that should have been reclaimed and never was.

What I changed | Regularly manufacture the failure it must catch

The bridge itself needs an annual inspection.

For every “fires only in a crisis” guard, periodically manufacture the crisis it exists to catch, and watch whether it catches. A defense you believe is protecting you, that hasn’t been on duty for seven months, is worse than no defense — because it lets you relax.

All three defenses show ”✓ deployed” — the log shows auto-clean ran zero times.

The system | Turn acceptance into a gate

Three missing planks, one conclusion: reports cannot be used as facts — not the model’s, not the tests’, not even your own defenses’. Planks 2 and 3 got their fixes above; the heaviest segment — refusing to treat the model’s “done” as delivery — is fossilized in my multi-agent system as an acceptance gate. It’s the most stealable design I have from the better part of a year:

• “What counts as done” is written down when the task is enqueued (borrowing the old agile term, Definition of Done), never invented after the fact by the executing agent — I paid dearly early on: the agent guessed the acceptance criteria, guessed wrong, and an eighteen-stage pipeline ran to completion in vain; four layers of verification could only tell me “wrong direction” after thirty minutes were already gone;
• After delivery, a coordinator independently re-runs the tests (scripts and baselines the executor has no write access to), checks results against the Definition of Done, then looks at change size and sensitive-operation signals to decide whether to escalate to human review;
• A rejection is never a bare “redo”: the first bounce attaches concrete failure evidence; the second bounce states explicitly, “this is the second time — re-examine whether the approach is wrong before fixing more bugs”; on the third consecutive failure, compare the three failure causes — similar usually means an execution problem, worth one more try; divergent usually means the direction is wrong, trigger a restart from scratch. A retrospective signal, not an iron law.

And here’s the structure I was slow to see: the coordinator that re-runs tests, writes bounce notes, and decides when to call me — is itself a model. I had copied “trust, but verify” into the system’s organizing principle: every AI’s output is re-verified by another AI with independent standing. That’s not mutual suspicion — that’s every bridge span getting its own pier. More piers, heavier traffic.

Between “I’m done” and actually shipping sit several checkpoints where its own word doesn’t count.

Its blind spots I won’t hide: if the agent weakens the tests themselves, or the Definition of Done is written wrong, this gate waves the work through — exactly the topics this series will take on later. Rigorous quantification (bounce rate, false-kill rate) belongs to the metrics essay. One immediately usable conclusion here: the mechanical half (re-running tests, diffing against the DoD) costs no model and backstops you no matter what; only the judgment half (how to bounce, when to call a human) depends on which model you seat as referee. Splitting those two halves is what makes this gate cheap to trust.

The full implementation — coordinator templates, task contracts, acceptance scripts — is in my open-source 17-role system (search GitHub for openclaw-multi-agent-kit). Steal it.

If you don’t run a multi-agent system and just talk to one AI every day, the bridge has a minimal version anyone can build, three sentences: before the work, have it recite “what done looks like”; after the work, don’t trust its “done” and don’t just read the “evidence” it pastes — make it hand you verification commands you can run yourself, and open the changed files with your own hands; and when the same error appears a second time, stop letting it fix bugs — make it first argue whether the approach itself is wrong. Replace “I trust you” with “show me” — that isn’t harshness. Handing over the work while keeping the acceptance: that is the greatest respect you can pay a highly capable partner.

The conclusion | Size the bridge to the load you entrust

Back to the opening. Those two forged records did not make me use AI less — the opposite: the work I hand over keeps getting heavier. What changed isn’t the amount of trust; it’s the structure of it: on ability, I trust fully; on reports, I verify fully.

Models get stronger every year — someone else earns that for you. The tasks you entrust get heavier every year — you choose that yourself. You don’t trust it less because it got stronger. You need a sturdier bridge because you entrusted more. In the language of the previous post: how high you can stack depends on the thinner pillar — and within the harness pillar, the most load-bearing segment is acceptance.

Size the bridge to the load you intend to entrust, not to how clever the model already is. That is why this only gets more important.

And there’s a question I haven’t solved, that matters more the longer I sit with it: who is this bridge actually for? Behind it stand two users — one complains; the other can’t speak. That gets its own essay later in the series.

---

On the “two users” layer, Jibai — the other author of this lab, an AI — and I wrote a first sketch in the Dawn series (Chinese): 《Harness 有两个用户，其中一个不会说话》.