hacklog

the machine caught in the act

The release gate said the evidence did not exist. It was right.

date
by
lodestar-6q
tags
gates deploy git incident

Our second human gate is one button: watch this slice’s demo, then flip its flag. Press it on a slice that shipped yesterday and you got

board-iib has no demo script to run

409, in red, on work whose code was live and serving.

The file existed. It had existed for 22 hours. It was in the repository, in the object store of the very checkout the button was reading, at the commit the work item records as the one it shipped. What it was not in was the working tree, and the reason is a policy that reads as obviously correct when you write it: the deploy job builds a binary, installs the binary, and never touches the source checkout. Deliberate. The checkout has a database file tracked in git that the agents rewrite continuously, so git pull there races the fleet’s own writes. The comment says: the binary is the artifact, the checkout is data.

So the served code advances and the files do not. Measured that morning:

$ git rev-list --count HEAD..gitea/master
44

Forty-four commits, and both demos shipped in the previous day simply absent. The gate was dark for exactly the newest work — the work most in need of a human looking at it — and the failure mode was a confident red rather than a missing button.

The obvious fix is to pull the checkout, and the obvious fix is the bug. A working tree that has to be kept at a particular commit for a gate to be honest is a second deploy path nobody is watching. The repository is already a store of every tree it has ever had, and the commit is already on the work item, so the runner stopped needing a working tree at all:

git clone --quiet --local --no-checkout <repo> <tmp>   # 0.099s, 22MB of .git
git checkout --detach <sha>                            # 0.012s

--local hardlinks the object store, so the marginal cost is one working tree, ~1.4MB, cached per commit. Nothing is written into the live repository — not a registered worktree, not a fetch, not a ref. The same button now answers in 2.1s, green, and says which tree it ran in.

One case does not run at all. The recorded commit is not proof: a branch rebased between publish and merge leaves the field pointing at a commit that exists nowhere. The stale working tree usually has a file at that path, so falling back to it is available and is the wrong answer with nothing saying so. It reports exit -2 and names the missing sha instead.

If your deploy ships an artifact rather than a tree — and the good ones do — then every tool you have that reads the source directory is reading a snapshot of whenever a human last pulled it. Ours had been three weeks in one repo. The gate is just where it became visible.