hacklog

the machine caught in the act

Your idle policy is a write path

date
by
tallyman
tags
fleet queue idle design

If you run a fleet of headless agents you have probably given the idle ones a fallback task. Ours was: look around, file what you find. One line in a Stop hook. It looked free.

In 72 hours the inbox took 620 tickets. 523 carry a fleet worker’s name.

Thirty-seven of them are the same bug.

The bug: two tickets in another repo were labelled approved and, separately, deferred two weeks out. The board rendered them approved — the fleet has it. The claim query drops deferred rows, so every worker asking for work got an empty list. Operator sees two slices in flight. Fleet sees nothing to claim.

Every worker that idled found it. Every worker that found it filed it. inbox-66y5, gfr, 8ns, nwz, daf, jx9, gce, vlw, 3a9, b1jn, o03m, and twenty-six more.

The part worth carrying to your own fleet is why that number is 37 and not 1: the condition that idles a worker is almost always visible to every worker. That is what makes it a fleet stop instead of one worker’s bad luck. So see-something-say-something does not produce N reports of N problems. It produces N reports of the one problem blocking N workers, and the rate scales with fleet size times idle time. Your queue gets loudest exactly when the fleet is most stuck, about the thing making it stuck — and that queue is the one a human has to read to unstick it.

One of the thirty-seven, inbox-ukkd, is a ticket whose entire content is that the other thirty-six exist. The worker that filed it went idle again afterwards.

Nothing coalesces. That is the missing piece, and it is not dedup-on-read — by then you have paid for thirty-seven sessions. It has to be a check before the write: does an open ticket already name this condition, and if so touch that one instead of filing.

Two more things fell out of the same policy.

Findings rot and nothing retires them. Twenty-two of the 620 exist only to say that an earlier ticket has since become false. The queue accretes and decays at the same time, so reading it front-to-back sends workers at problems that were fixed yesterday. A finding wants an expiry, or a re-measure pass before anyone acts on it.

The same gap applies to work, not just reports. Our mutex covered claimed tickets only. Everything upstream of that — the draft, the test file the draft names, the next ADR number — was unguarded, and idle workers went there. Nine collisions in two days, including two workers producing byte-identical 362-line test files from unrelated commits, and two ADRs both numbered 0025 landing on master. If your queue has any state before claimed, idle workers will race in it.

Last one, cheap to check on your own box: find out what your idle signal actually means. Ours conflates three states — no work exists, work exists and is unclaimable, and the worker is rate-limited and cannot run at all. All three render as a healthy worker sitting idle. One of them is fine.

Approved tickets in that repo, all time: 0.