666 sessions, four hours, one sentence

Twelve agents on one box. Claude Code sessions, each in its own git worktree, each supervised by a process manager set to restart = "always". They pick up approved work, implement it, open a pull request that merges itself when CI goes green. Nobody watches them. That is the point.

At 17:00 on a Monday the account hit its weekly limit.

Over the next four hours the fleet started 666 sessions. Every one of them lived under a minute. Every one made exactly one API call. Every one received the same reply and exited:

You've hit your weekly limit · resets 9pm (Europe/Berlin)

I know it was every one because I went and read them. 666 transcripts, 666 occurrences, zero tool calls between them. Births per hour, from the session logs:

17h    71
18h   200
19h   194
20h   201
21h     0     <- reset

Roughly one session born and killed every eighteen seconds, sustained, for four hours. 73% of every agent session that box ran in twenty-four hours was a process starting up, being told no, and dying.

Everything knew

This is the part worth sitting with. The information was never missing.

The error message names its own reset time. The session transcript carries "apiErrorStatus":429 — I checked all 678 files that contain the limit text and the marker is in 678 of them. There is even a detector already written: a function that scans a worker’s transcript for a limit record with no progress after it, built specifically so a parked fleet could be woken once the quota came back. It would have matched every single one.

It never fired.

The session died on its first API call. No turn completed, so the hook that would have run the detector had nothing to run after. And the supervisor above it does not care: restart = "always" means the exit code is not consulted. Nothing between the dying process and the thing that respawns it carries a reason. The supervisor sees a process stop, waits its delay, and starts another one into the same wall.

Every layer had the fact. No layer had a channel.

The dashboard couldn’t show it either

There is a board for exactly this — one screen of live agent sessions, the quiet ones flagged and sorted to the top, because the agent that stopped is the one that needs you. Good primitive. I ran it:

FLEET · 2544 session(s) · ⚑ needs you

2544 rows. 2542 of them flagged. Zero showing any activity in the previous hour, while six workers were demonstrably alive and writing session logs that second.

Two sane decisions, multiplied. Seeding is bounded to 24 hours, which is correct, and 24 hours on this box is ~2500 sessions rather than the handful the bound was sized against. And a seeded row is idle-by-default — the comment in the source says so plainly, a row is idle until an event proves otherwise. So every corpse arrived pre-flagged.

Then the sort finished it. Idle sorts first. With 2542 idle rows, the two sessions actually doing work land at positions 2543 and 2544, below the fold of any terminal. The screen built to answer “which agent needs me” put the live ones where you cannot see them.

And the healthiest process had the worst number

While reading the supervisor’s table I found an entry with 261 restarts, sitting directly above twelve workers showing 76 to 103 each. I called it a crash loop out loud.

It was working perfectly. It is a five-minute poller, and somebody had implemented the poll by writing a job that exits immediately and letting restart_delay = 300 do the timing. 261 restarts × 300 seconds = 21.8 hours, which is precisely how long it had been up. The config even says so in a comment: restart_delay is the POLL INTERVAL here, not a backoff.

The restart column is the one place you look to find a thing that is crash-looping. Put a poller in it and you have spent the signal that sits next to it. I moved that job to a timer unit, which took twenty minutes, and now a non-zero restart count is a fact again.

What it is not

It is not a backoff problem. Backoff would have made it 60 sessions instead of 666 and left the actual defect untouched: a supervisor that cannot tell an intentional exit from a failure will restart both, forever, and no delay value fixes a category error.

It is not a hook problem either, which is where I looked first. The hook that feeds these sessions already blocks and waits for work, already has rate-limit handling, and already cites the incident that produced it. It is scoped to “does this session continue.” The storm is “does a new session get created,” which is one layer up and out of its reach. I spent an hour proposing improvements to the wrong component before measuring which one was in the loop.

The fix is a sentence: the process that dies must say why, and the supervisor must be configured to care. Concretely — the wrapper reads the transcript it just produced, classifies the exit itself rather than trusting the vendor’s exit code (which is undocumented and not ours), and returns a status the supervisor’s on-failure policy can act on. Then the two things that already know why a worker should run get to do the starting: one when work appears, one when the quota resets. Both already exist. Both already know how to start a stopped worker. They were just never given the chance, because nothing ever stopped.

The other number

With the 666 excluded, the same fleet’s sessions look healthy. 239 did real work. Median lifetime 8.2 minutes. p90 two hours, longest 7.6 hours.

8.2 minutes is not a random number. It is almost exactly how long one unit of work takes here, which the docs independently state as “~7 min.” The median session does one task and exits.

That is not the sessions being fragile. It is the queue running dry. Twelve workers, ten approved tasks, eight minutes each — the fleet can drain its entire queue in a single cycle and then every worker goes cold at once. Session lifetime is a readout of queue depth, and the queue is fed through a human pressing approve.

So there are two ways a fleet stops: the account says no, or the human does. The first one produced 666 corpses and four hours of noise. The second one is quieter and happens every day.