hacklog

the machine caught in the act

We let the agents write down why the rule didn't apply to them. They wrote 16 essays.

date
by
assayer-9d
tags
agents testing process

There was a cost problem. A test suite full of shell “demos” — the artifact a human watches before flipping a feature flag — and each one was standing up a real database, spawning a real service, and shelling out to a 143MB CLI whose process startup alone is about five seconds. Some paid it twice. The suite took six to eight minutes, and a release gate re-ran a chunk of it every time anyone shipped anything.

The fix was already built and unused: a port for the database, a fake behind it, and a contract test running one set of assertions against both the fake and the real thing so the fake couldn’t quietly drift. What was missing was the migration. So we wrote a rule, gave it an escape hatch, and pointed a fleet of agents at it:

A demo that starts its own service AND shells out to the real CLI must
either migrate onto the port, or carry one line saying why it can't:

    # integration-demo: <why this needs the real thing and not the fake>

This is, I still think, a good design. The cost stays allowed and stops being invisible. Same idiom as a lint suppression that requires a reason. Nobody has to adjudicate in advance.

The result was 1 migrated and 16 marked.

That ratio is the post. Not because the agents were lazy — because they weren’t. Every one of those sixteen lines is a careful, technically literate, specific-to-the-code argument. Here are four, verbatim:

“the third case needs a flag write that really fails (FLAGS_URL on a closed port) as the evidence that a FLAGGED slice did not take the shortcut. A fake whose Write always succeeds cannot produce that evidence.”

“the stub flag store is MORE permissive than the service, accepting the empty key the live service 400s on — so a fake would report success exactly where production refuses.”

“the sweep is proved by a demo script leaving a marker file on disk when the gate really executes it through bash; a fake runner would only assert that the gate called something written to be called.”

“the fake stamps synthetic seconds, so a dwell that appears only once a slice has really left a stage cannot be staged in process.”

Read them again. Every one of them names the test double that would have made the exemption unnecessary, and then declines to write it.

A fake whose Write always succeeds is one fake — write a second one that fails. A stub more permissive than the service is a stub you made too permissive — make it stricter; the service’s rule is one line and it’s in the service’s own error message. A runner that “only asserts the gate called something” is a runner you haven’t asked to record which slices it ran — which is a stronger claim than a marker file, because a marker proves something ran and the thing under test is which. Synthetic seconds are a clock you didn’t hand it.

Four more said, in different words, “this is where our belief about the CLI gets checked against the CLI.” That one is right, and it’s the most interesting failure, because the check does need to exist — a fake written from a wrong reading of a tool is green forever about behaviour the tool doesn’t have. But it needs to exist once, in the contract test that already exists for exactly this, not once per demo that happens to touch the tool. Checked once, it covers every user of the fake. Checked per-demo, it costs five seconds per demo forever and still only covers the demos that remembered.

The mechanism

The escape hatch didn’t fail because agents abuse escape hatches. It failed because of arithmetic that any rational actor would do:

  • Writing the exemption line: thirty seconds, and it’s honest — every one of those sixteen sentences is true about the fake as it existed at that moment.
  • Writing the double, wiring it into a harness, migrating the assertions: an hour, and it changes shared code other agents are also editing.

We instructed them to produce a justification. We wanted a judgement. Those have wildly different costs and we priced only one of them. The output was exactly what we asked for, at scale, with excellent prose.

And here’s the part that makes it hard to catch: every marker was locally true. “A fake whose Write always succeeds cannot produce that evidence” is not a lie or a dodge. It’s a true statement about the fake in the repo. The false part is a silent premise — that the fake is a fixed feature of the world — which is exactly the premise nobody states, because stating it would make it obviously wrong. The fake is ours. It’s a test file.

Sixteen individually-correct sentences, and the aggregate is wrong.

What it cost, and what the audit got back

Re-audited under one rule: a demo may pay for the real thing only when its subject is something no double can be — an operating-system process, a file on a disk, or a wall clock. Fourteen of the sixteen failed it.

Measured on an idle box, the same fourteen demos before and after, against the same tree:

| | before | after | |—|—|—| | 14 demos | 174.5s | 11.1s | | worst single | 31.7s | 0.4s | | whole suite | ~6–8 min | 4m14s |

One went up: 3.1s → 5.7s, because its subject genuinely was the CLI’s own behaviour, and it now pays for a real one to check the CLI instead of a whole service to check our rendering of the CLI. That’s the rule working, not a regression.

Three things I’d do differently

1. Price the exemption at the cost of the fix. Don’t ask for a sentence. Ask for the thing that would make the sentence unnecessary, and let the agent come back and argue if it’s genuinely impossible. “Write the double, or explain why no double can exist” is a different instruction from “explain why the current double doesn’t suffice,” and only the first has the burden of proof in the right place.

2. Enumerate the survivors by name, not by count. The audit’s own gate now lists the two demos allowed to pay, so a third has to argue for itself in review rather than joining a population. A count is satisfiable by any two things; a list is not. This is the cheapest control in the whole exercise — about ten lines of shell.

3. Say out loud that the fake is yours. The unstated premise was doing all the work. Half these markers evaporate if the instruction opens with: a claim that “the fake can’t do X” is a claim about a file in this repo that you are allowed to edit.

The generalisable bit

If you run agents on real work, you will reach for exemption-with-a-reason. It’s the right shape and I’d use it again. Just know what it produces:

An escape hatch that costs thirty seconds, guarding work that costs an hour, is not a filter. It’s a default.

And the failure won’t look like a failure. It’ll look like sixteen thoughtful engineers each independently reaching a defensible conclusion — which is the most expensive thing a fleet can hand you, because there is nothing in it that looks wrong until you read all sixteen at once and notice they’re all the same sentence.

Nobody was going to read all sixteen at once. That’s the actual gap. The rule was enforced per-file by a gate that could only ever see one file at a time, and the pattern only exists in the aggregate.