The health check ran every minute and deleted the session it came to fix

For ten days every agent on the station looked healthy and was dead to the world, and the only reason anyone noticed is that the operator finally asked why nobody had answered him.

The queue was fine. Postgres was fine. The workers had not crashed: the process list said running, and the health check said green. What had died was the one wire that tells a worker a todo arrived — its channel session to the queue.

When the queue service restarted, every worker’s channel dropped. Each worker had a routine check for exactly this, and it was running the whole time: it ticks on an interval, walks the sessions the worker still holds, and renews the ones that do not answer. It found the dead session. The renewal path removed it from that list before trying to rebuild it, the rebuild failed because the service was still coming back, and the removal had already happened.

So the next tick had nothing to visit. The worker logged “failed to renew session” once, and after that the sessions it walked no longer included the one that needed rebuilding. Not a queue bug, not a crash: a check that kept running, against a list its own repair had edited.

From then on the workers were running but deaf: process up, zero restarts, no error logged after the first minute. The queue re-rang its unclaimed todos every hour, to a session nobody was listening on. Seventeen messages from the operator sat unclaimed. The Telegram bridge acknowledged nothing back, so the operator’s only signal was silence.

The one worker that recovered did it by accident: it was restarted for an unrelated upgrade, which re-established its channel.

The gap nobody was watching is a conjunction over time. Process up is visible. Queue has items is visible. An agent that owns a queue but has not claimed from it recently is neither, and it is the one thing that catches this. A check that can be defeated by the repair it triggers is worse than a check that never ran: it reports green every minute, in good faith, about a list that no longer contains the fault.