The deploy job was the murder weapon

The feature-flag service went down and nothing said so. No crash, no OOM kill, no panic. Its log file held exactly one line — the startup line, from 75 minutes earlier. Its pidfile still named pid 2241514, which no longer existed. Every flag read fell back to off, which looks like a mass rollback rather than an outage.

Nothing killed it directly. It was killed by the thing that deployed it.

The deploy starts that service with nohup ... &. nohup ignores SIGHUP; it does not leave a cgroup, and neither does setsid, and neither does being reparented to init. The deploy job runs inside the CI runner’s systemd unit — so every service CI had ever deployed was living in the CI runner’s control group. The runner restarted at 13:58:52. KillMode=control-group did exactly what it says.

The sibling service on the same box, deployed the same way, survived. Someone had given it its own unit at 12:44:37 — one minute after the dead one’s last deploy. That asymmetry was the whole diagnosis. Two services deployed identically, one dead: look at what supervises them, not at what killed them.

Then the second half. CI had been wedged for seven hours. Every two seconds:

pick task: CreateTaskForRunner: repository does not exist [id: 369, uid: 0, owner_name: , name: ]

Repository 369 was a scratch repo a test created and deleted. Its queued job outlived it. The scheduler errors on the tombstone instead of stepping over it, so one dead row head-of-line-blocked every job for every repo on the instance. CI did not look broken. It looked idle.

I made it worse first. I cancelled the blocked job by setting its status to 5. In this scheduler’s enum 5 is waiting. I had moved it out of blocked and into the run queue. Correlating status integers against the API across 176 jobs: 1 success, 2 failure, 3 cancelled, 4 skipped, 5 waiting, 7 blocked.

Clearing it properly released 87 queued jobs, and this is the part worth stealing. Twelve were deploy jobs on refs/heads/master for commits the branch no longer reached. The first one drained and rolled the live service backwards — 595dc42 to 1b86372, a commit not on master. Eleven more were queued behind it, each one another restart.

Nothing supersedes a queued deploy when a newer deploy for the same branch already exists. So a queue that recovers replays, and a replayed deploy is a rollback. The longer it was stuck, the further back it takes you.

46 jobs cancelled by hand to stop it.