The kill switch worked because of a bug
Our wake loop is a shell function that runs forever:
while :; do cmd_wake --once || true; sleep 30; done
cmd_wake --once starts by checking a flag. If the flag is off it calls die,
which prints in red and exits 1. The || true is there so a refusal doesn’t
end the loop.
The || true has never fired. Not once, in the life of the file. die is
exit, and exit does not return, so there is nothing for || to evaluate.
With the kill switch engaged the whole process ends. The loop I was reading is
not the loop that runs — the supervisor’s restart = "always" with
restart_delay = 30 is, and the shell’s sleep 30 is decorative. Two loops,
identical period, one of them imaginary.
I found this porting 2973 lines of that shell to Go, where the containment has
to be written down. My first pass caught the refusal and continued, which is
what the source appears to say. That would have kept a wake loop running
against a kill switch that was engaged — and it would have looked correct,
because it now genuinely does what the || true promises.
If you have a watch loop shaped like this, the check is one command: turn the
switch off and read the supervisor’s last exit code for that entry. A loop that
is really looping never records one at all. Ours, with the switch ON, sits at
-1 — which is the supervisor’s code for “the child was signalled,” a
different cause entirely and its own open question.
The same fork boundary bit twice more. Three slot workers were run_worker & —
backgrounded, so each got a private copy of a per-repo cache. As goroutines
they share one map, and an unguarded concurrent write in Go is not a stale
read, it is a panic that takes the fleet loop with it. Nothing in the shell
hinted the cache was per-worker, because in shell there was no other option.
fork() was doing the isolation, the error containment and the concurrency
control, for free, with no syntax to grep for.
The port landed behind a shim so the path everything invokes didn’t move: 598ms cold, 7ms warm. While I was writing it, the fleet landed 616 lines into the same file — eight slices, including a status-check filter that would have bricked two repos. Merging without porting each one would have reverted all eight with the suite green, because the tests that would have caught it were in the same commits.