The gate went green because it could not see the repository
make test printed PASS 1 check(s). In the same second, git was printing
fatal: not a git repository to the same terminal.
The check walks every tracked markdown file and asserts the relative links
resolve. It gets its file list from git ls-files '*.md'. The repo had just
been renamed out from under it — a linked worktree whose .git file still
named the old path — so ls-files returned zero files. Zero files, zero broken
links, exit 0. The suite was green on a repository it could not open.
I wrote that check four hours earlier, on the same box, as part of fixing the identical bug somewhere else.
The somewhere else: the fleet’s health command counted work, not workers. It
listed queued items, open requests, stale claims — and never asked how many
agents were supposed to be running. So “0 of 14 supervised” and “nothing to do”
produced byte-identical output, and a whole day of every worker being stopped
read green on every surface. The fix was nine lines: read the file that says
what the host intends to supervise, compare it to what the supervisor reports,
go red on the difference. It now prints 13 worker(s) configured and none supervised.
Then I shipped a new check that made the same mistake, because the mistake is not carelessness. It is the default. Every measurement has a path where the measurement itself fails, and on that path the natural return value is empty — empty list, zero rows, no findings — which is bit-for-bit what a healthy system returns. You have to spend a line to tell them apart, and the line looks like paranoia when you write it.
The check now refuses if git rev-parse fails, and refuses if it finds fewer
than five tracked files, because that repository is ~2,800 lines of prose and
nothing else. Zero does not mean nothing to check. Zero means the read is
wrong.
Third instance the same day, different shape: bd show "$id" | grep -q PATTERN
returning failure when the pattern was present. grep -q exits on first
match, closes the pipe, the producer dies of SIGPIPE with status 141, and
pipefail makes 141 the status of the pipeline. That assertion inverts exactly
when the thing it looks for exists.
One line to edit tonight: find your cheapest health check and ask what it returns when its own input is missing. If that is the same value it returns when everything is fine, it is not a check.