Don't ask the sick process how it feels

For about a day, the worst problem in my agent fleet was that a dead worker looked exactly like a busy one.

The setup: long-lived agent sessions, one per worktree, each fed by a hook that claims the next approved task when the session would otherwise stop. When the account’s session limit gets hit, every one of those sessions stays alive. The supervisor lists them as running. The health check returns green. And nothing is claimed, for hours, because every attempt to actually do work dies on the rate limit. The hook doesn’t know why a session is stopping, so it cheerfully hands work to a session that cannot run it, and the failure becomes a tight loop — in one worker’s log I counted ~50 “Execution error” strings against 19 “session limit” lines.

My instinct, and I think most people’s, was: I need a health signal. Get the worker to report its rate-limit state, thread that through the supervisor, condition the scheduler on it.

That instinct is wrong, and it took me embarrassingly long to see why.

The failure mode that matters is the one that leaves the reporting path intact. A crashed process is easy — it’s gone, and its absence is the signal. What actually eats your day is the process that is alive, responsive, correctly answering “am I up?” with “yes”, and structurally incapable of doing its job. Rate limits do this. So does an expired token, a wedged tool call, a full disk the process hasn’t tried to write to yet, and a queue consumer whose connection dropped in a way the client library is quietly retrying forever. In every one of those cases, the health endpoint you built is being served by the part of the system that isn’t broken.

So conditioning the alert on worker self-reports makes it silent in precisely the case it exists for.

Watch the work instead. I have a board that already knows three things about my fleet, none of which comes from the workers:

• is there work available to claim
• is there idle capacity
• has anything moved recently

Available work, idle capacity, and no state transitions for N minutes means the fleet is dead, no matter what the fleet says about itself. I don’t need a new signal. I needed to stop asking the patient.

This generalises past agents. If you run anything that pulls from a queue, you probably already have the queue depth, the worker count, and a timestamp on the last completed item. Those three, in conjunction, are a better liveness check than any heartbeat the worker emits, because they are measured on the other side of the thing that might be lying.

Two things I’d get wrong if I built this quickly.

First: the alert has to carry its numbers. “Fleet stalled” is a sentence you disbelieve the first time it’s wrong, and after that it’s wallpaper. “Fleet stalled: 6 claimable, 3 of 6 slots idle, last transition 47m ago” is a sentence you can check in ten seconds. An alert nobody can check stops being an alert and becomes a notification.

Second, and this is the subtle one: a monitor built on absence has to be able to tell “I saw nothing” from “I couldn’t see.” My board reads one database per repository. If one of those reads fails and the failure gets swallowed — which is exactly what a return nil, nil error path does — the depth drops, the queue looks empty, and “no work available” reads as a healthy idle fleet. The alert that exists to catch a stalled fleet gets suppressed by a broken read, because both produce the same zero.

So if you build this: make the partial read loud. A number derived from an incomplete scan must be labelled incomplete, everywhere it surfaces. Otherwise you’ve built a monitor whose blind spot is the failure of its own eyes, and it will be confidently green on the day you most need it.

The whole thing collapses to one line, which I’ll be repeating to myself for a while: health is not what a process says about itself, it’s whether work is moving.