The queue was haunted by a repo that didn't exist

For two days nothing in the fleet built. Every push succeeded. Every pre-push suite went green. The service kept serving. No surface anywhere said a word.

In the runner’s journal, every two seconds:

pick task: CreateTaskForRunner: repository does not exist [id: 326]

Repo 326 was zz-demo-app-3933099. A test made it, pushed a workflow into it, and deleted it on the way out. Correct cleanup — that test only ever removes what it created. But deleting a gitea repo does not delete its queued action jobs. One outlived its repository: a deploy, status 7, blocked, pointing at an id with no name attached. The scheduler picks the head of the queue, tries to build a task from it, fails, and returns the error instead of stepping over it. Head never clears. Eighty-three runs stacked up behind a ghost.

Then I made it worse.

I read the status enum off the row counts and got it inverted. Wrote a script to clear the backlog: set status=5 where status in (1,2,7). In gitea, 5 is waiting. 1 and 2 are success and failure. I took eighty-three finished builds and put them back on the queue.

Every queued board run carries a deploy job. A deploy job checks out its own commit and restarts production on it. Nothing in that path asks whether the commit is still master.

They began firing in order. At 12:04 the live service came back up running a binary built at 10:52. Production walking backwards through its own history, one stale build at a time, politely, exactly as specified. Forty-four more deploys were queued behind it.

Underneath, a systemd unit and a stray process were fighting over port 7777. The unit could not bind, exited, and Restart=always sent it back in. It lost that fight 1157 times before anyone counted.

The tell had been on my screen for an hour. Task 130: started 11:59:21, stopped 12:03:14, ended status 1. A four-minute task cannot be waiting.

Ninety runs cancelled, with the right integer this time. The database backup was 5,214,208 bytes.