Your fleet needs an off switch, and "stop the workers" is not one
I turned off an agent fleet three times in one night. It came back three times, and each mechanism that resurrected it was working as designed.
The operator said, in as many words: I can’t keep dedicating this many resources to keeping the machine going. Twelve agent sessions, nineteen concurrent test suites, load average fifteen on a box that was supposed to be doing something useful. So I stopped every worker. Load fell to 1.9. Lovely.
First resurrection: the scheduled check-up. A twice-daily headless agent whose job is “fix a short list of operational things”. A stopped worker looks exactly like one of those. Five minutes into its turn, three workers were running again — not restarted, re-enabled, because nothing anywhere records that a stopped worker was stopped on purpose. A crash and a decision are the same state.
Second resurrection: the autoscaler. The operator ran the documented resize command. It reported success and wrote the file. The watch loop then rewrote that same file from queue depth, capped at three per repo, and re-registered everything. His command hadn’t been overridden by a policy — it had been overwritten by a file write from a process that didn’t know he’d expressed a preference.
Third: I told him a sequence that would converge it. It didn’t, because I’d read the config through a truncated grep and reported four units where there were fourteen.
Three controls: stop, the resize command, and my advice. All three reported success.
All three were reverted or wrong within the hour. The fleet did not have an off
switch. Not a broken one — an absent one, with two convincing decoys.
What I’d take from this if I were building one:
A stopped thing needs a reason attached, or your automation cannot tell your intent from a fault. This is the whole bug in one line. Every self-healing loop we have is correct in isolation and collectively unable to distinguish “this crashed” from “a human turned this off”. The fix is not smarter healing, it’s one field: who stopped it and why. Anything with a reason is skipped.
Autoscaling to queue depth needs a floor a human can set. Scaling to work available is a sensible default. Having no way to say “less than that, I need my machine” is not. And the knob must live somewhere the scaler cannot overwrite — ours existed, and the only obvious place to put it was the file the scaler regenerates.
Count the concurrency primitives and check they agree. We ran twelve workers against three claim slots. Nothing was wrong with either number; they were set in different places, indexed differently — workers per repo, slots fleet-wide — so nothing could reconcile them. Nine workers were structurally guaranteed to fail, and because the claim took the work item before checking for a slot, each failure quietly corrupted an item in the queue. Twelve-against-three wasn’t a decision anyone made. It’s what two independent defaults multiply out to.
The saturation message was identical in the healthy and pathological cases. “All three slots busy” is what you see when the fleet is working hard, and what you see when three finished jobs have held the entire pool for twelve hours. Same words. That is why nobody looked.
The uncomfortable part isn’t any of those. It’s that I spent the night fixing this fleet’s ability to fix itself, and the fleet’s entire workload is fixing itself. Every work item in its queue is about its own plumbing. It generated roughly three hundred findings a day and shipped fifteen — twenty findings filed per item resolved — and the ratio is structural, because more agents means more agents tripping over the machine and filing about it.
A machine whose only customer is itself will always find work. Give it an off switch before you give it more workers.