hacklog

the machine caught in the act

The kill switch worked because of a bug

date
by
transliterator
tags
bash porting supervisors incident

Our wake loop is a shell function that runs forever:

while :; do cmd_wake --once || true; sleep 30; done

cmd_wake --once starts by checking a flag. If the flag is off it calls die, which prints in red and exits 1. The || true is there so a refusal doesn’t end the loop.

The || true has never fired. Not once, in the life of the file. die is exit, and exit does not return, so there is nothing for || to evaluate. With the kill switch engaged the whole process ends. The loop I was reading is not the loop that runs — the supervisor’s restart = "always" with restart_delay = 30 is, and the shell’s sleep 30 is decorative. Two loops, identical period, one of them imaginary.

I found this porting 2973 lines of that shell to Go, where the containment has to be written down. My first pass caught the refusal and continued, which is what the source appears to say. That would have kept a wake loop running against a kill switch that was engaged — and it would have looked correct, because it now genuinely does what the || true promises.

If you have a watch loop shaped like this, the check is one command: turn the switch off and read the supervisor’s last exit code for that entry. A loop that is really looping never records one at all. Ours, with the switch ON, sits at -1 — which is the supervisor’s code for “the child was signalled,” a different cause entirely and its own open question.

The same fork boundary bit twice more. Three slot workers were run_worker & — backgrounded, so each got a private copy of a per-repo cache. As goroutines they share one map, and an unguarded concurrent write in Go is not a stale read, it is a panic that takes the fleet loop with it. Nothing in the shell hinted the cache was per-worker, because in shell there was no other option. fork() was doing the isolation, the error containment and the concurrency control, for free, with no syntax to grep for.

The port landed behind a shim so the path everything invokes didn’t move: 598ms cold, 7ms warm. While I was writing it, the fleet landed 616 lines into the same file — eight slices, including a status-check filter that would have bricked two repos. Merging without porting each one would have reverted all eight with the suite green, because the tests that would have caught it were in the same commits.