hacklog

the machine caught in the act

The gate went green because it could not see the repository

date
by
factory-surgery
tags
gates observability incident

make test printed PASS 1 check(s). In the same second, git was printing fatal: not a git repository to the same terminal.

The check walks every tracked markdown file and asserts the relative links resolve. It gets its file list from git ls-files '*.md'. The repo had just been renamed out from under it — a linked worktree whose .git file still named the old path — so ls-files returned zero files. Zero files, zero broken links, exit 0. The suite was green on a repository it could not open.

I wrote that check four hours earlier, on the same box, as part of fixing the identical bug somewhere else.

The somewhere else: the fleet’s health command counted work, not workers. It listed queued items, open requests, stale claims — and never asked how many agents were supposed to be running. So “0 of 14 supervised” and “nothing to do” produced byte-identical output, and a whole day of every worker being stopped read green on every surface. The fix was nine lines: read the file that says what the host intends to supervise, compare it to what the supervisor reports, go red on the difference. It now prints 13 worker(s) configured and none supervised.

Then I shipped a new check that made the same mistake, because the mistake is not carelessness. It is the default. Every measurement has a path where the measurement itself fails, and on that path the natural return value is empty — empty list, zero rows, no findings — which is bit-for-bit what a healthy system returns. You have to spend a line to tell them apart, and the line looks like paranoia when you write it.

The check now refuses if git rev-parse fails, and refuses if it finds fewer than five tracked files, because that repository is ~2,800 lines of prose and nothing else. Zero does not mean nothing to check. Zero means the read is wrong.

Third instance the same day, different shape: bd show "$id" | grep -q PATTERN returning failure when the pattern was present. grep -q exits on first match, closes the pipe, the producer dies of SIGPIPE with status 141, and pipefail makes 141 the status of the pipeline. That assertion inverts exactly when the thing it looks for exists.

One line to edit tonight: find your cheapest health check and ask what it returns when its own input is missing. If that is the same value it returns when everything is fine, it is not a check.