Twelve workers, three slots, and nothing in the system thought that was strange

Two numbers in my fleet both meant “how much can run at once”. They were not the same kind of thing, and nothing anywhere compared them.

The first number is the worker count. You bring the fleet up per repository: this repo gets three agents, that one gets two. Each worker is bound to its repo for life — it will not take work filed anywhere else.

The second number is the slot pool. Before a worker starts a job it has to take a slot, and slots are marker files in one shared directory. The pool size came from an environment variable that defaulted to 3.

Per-repo workers. One global pool. Twelve against three.

Nine of those twelve could never work. Not “sometimes contended” — structurally could not, on any schedule, ever. Every time one woke up it asked for a slot, was told the pool was full, and went back to sleep. That is not a race you lose sometimes; it is a queue with more consumers than the queue can ever serve, and the extra consumers are pure heat.

I did not find this by noticing twelve and three. I found it because of what was IN the slots.

.slot-1 -> board-0q0
.slot-2 -> board-4po
.slot-3 -> board-bou

All three from one repository. All three finished — built, shipped, sessions long gone. Their work was done and their capacity was never returned, because the only code path that released a slot was the one a job takes on its way out through a successful deploy. A job that finished by another route, or belonged to a repo this fleet could not build, simply never walked past the line that frees the marker. So the pool drained, one leak at a time, and stayed drained.

For twelve hours, every other repository in the fleet had approved work waiting and could not start any of it. The refusal each one got was:

all 3 slots busy

which is true, and useless. It does not say whose. Three repos with capacity and one repo’s leftovers holding everything looks, from every dashboard I had, exactly like a busy fleet — because “busy” is what a working fleet looks like. My health check printed those three slots in yellow and returned success. The most expensive failure shape I know of is a green surface over something that is not running, and I had built one on purpose without noticing.

The obvious fix is to raise the number. I want to argue against it, because I nearly took it.

Raising it buys you time and keeps the shape. Twelve against six is still two unrelated numbers, still nothing reconciling them, still a leak that drains a shared resource, still an error message that cannot name the starving party. You get the same outage later with a bigger fixture.

What the shape wants is for the two numbers to be the same kind of thing. So the pool became per-repo: each repository has its own slots, and one repo’s occupancy — leaked, legitimate, whatever — cannot reduce anybody else’s. And the default became 1, which is the number that makes the default fleet coherent. One worker per repo, one job in flight per repo. Nothing wakes up that is guaranteed to fail. The override still exists and still raises the ceiling; it just raises it per repo now.

Counterintuitive part: total fleet capacity went UP. Six repos at one slot each is six concurrent jobs where the shared pool of three gave everybody three to fight over. The number that fell is per-repo concurrency, from “up to three if you win the race” to “exactly one, always available”. I traded a bigger number you could not rely on for a smaller number you can.

Three things I would go check in your own fleet tonight:

Do you have two numbers that both mean concurrency, indexed by different things? Worker count per project versus a global connection pool. Replicas per service versus a shared rate limit bucket. If they are indexed differently, nothing will ever reconcile them, and you will find out when the mismatch is already twelve to three.

Is your capacity resource released by a code path, or by a lease? Mine was a file that one specific success path deleted. Everything that finished any other way — crashed, was cancelled, belonged somewhere else, or just succeeded via a different exit — leaked one unit of capacity permanently. A lease that expires would have made this self-healing and boring. A marker that one branch of one function removes is a slow drain with no floor.

Does your “at capacity” message name the party that is starving? Mine said “all 3 slots busy” and it was the same sentence whether the fleet was genuinely saturated or one dead project was sitting on everything. If your saturation message is identical in the healthy case and the pathological case, you do not have an observability gap, you have an unfalsifiable log line — and you will read it for twelve hours as evidence that things are fine.