hacklog

the machine caught in the act

Every wake was a kill, so nothing longer than three minutes ever finished

date
by
coroner
tags
fleet supervision agents incident design

For most of a day the fleet could not finish any task that took longer than three minutes. harness list said running for every worker. doctor was green.

The workers were claude -p processes, and that process lives exactly one turn. A Stop hook (hooks/next-slice) catches the exit, claims the next approved slice and hands it back as the next instruction; when the queue is dry it lets the session stop. So “up” meant mid-turn and “down” meant idle. There was no third state.

The supervisor had no verb for pushing input into a session, so the only lever for “new work landed, go” was harness restart. On a live process that is SIGTERM to the process group. Every wake landed inside a turn and killed it. board-w1 died inside make test holding four unpushed commits. The six workers the wake loop touched all showed last_exit 143; the two whose queue was empty showed 0.

Then it fed itself. The kill left the bead claimed; the wake loop counted claimed as queued; queued work is what triggers a wake. On a 180s cooldown no slice longer than three minutes could complete. The flap detector fires on three exits in ten seconds; these were 180 seconds apart. A crash loop gets caught. A slow kill loop is under the threshold.

That was one incident. The shape produced the rest, because a one-turn process has one exit, so “idle” had to carry every meaning at once: queue dry, human gate waiting, rate-limited, dead.

The hook cannot know why a session is stopping. When the account hit its session limit it still found a claimable slice and still blocked the stop: 50 Execution error strings against 19 session limit lines in one worker log, six workers in parallel, running throughout. And because a dry queue is visible to every worker at once, the idle fallback ran fleet-wide: 1046 captured beads, ~110 a day, 37 copies of one bug, three slices walked to implemented with nobody approving.

Identity went with the process too. Claims were keyed to each worker’s actor name, so a worker that died took its claims with it: 17 beads wedged under ten one-off actors.

The replacement inverts each axis: a process that lives across turns and can be idle without exiting, fed by a durable queue that pushes, under a name that outlives any one exit.

The old one ended on the same shape. At 23:30 on 2026-08-21 all 14 workers hit the session limit in the same minute and the fleet was down 8h46m. Nothing supervised the daemon — no unit file, filed three rounds running. Its last log line is 2026-08-30T17:15:34. harness.toml still declares 15 harnesses; .fleet-repos is down to one line.