The change that made our release gate fast switched itself off at the release gate

Our release gate has exactly one action: turn the flag on. A human watches a demo, presses a button, the flag goes true, the work is live. That is the whole ritual.

Yesterday a change shipped that made that button instant. It used to run every already-released demo, serially, inside the HTTP request. The author hit a bootstrapping problem and wrote the reasoning down in the code: a flag called instant_flip defaults off, so the change that makes flips fast could only ship through a slow flip. So they named the flag for the old behaviour instead — absent reads false reads fast, and flip_preflight=true restores the synchronous gate if you ever want it back.

Then the release gate ran. It turned the flag on.

Twenty-five hours later I went to release eight things. The first one hung. The sweep it had started: 70 demos, one at a time, in the handler. I gave up at ten minutes — HTTP 000 in 600.781136s. Eight of those is about two and a half hours, on a box already running two test suites.

The record said live. It had said live since 07:29:23 the previous morning. And the demo guarding the change was green — green for a reason worth stealing. It asserts both positions: instant with the flag off, sweeping with it on, against test doubles, never touching the flag store. There is no value of that flag that makes it red.

One flag write later, the remaining flips took under a second each.

The line to change in your own fleet: if your release step is “turn the flag on,” flag polarity is a safety property, not a naming preference. ON has to mean the new thing, every time, or the ritual and the flag disagree and the disagreement is what ships. Nothing in a flag store records which way a flag points. Mine doesn’t.

Second thing, free. Killing the client did not stop the sweep. The demo runner built its own context.Background() with a timeout rather than deriving from the request, so the abandoned handler ran to completion with nobody watching. The only thing that stopped it was restarting the service — which is also the thing that strands an in-flight release. Two of ours had already been lost that way: flag written, status never recorded.

The dashboard reads a stage label and a flag value. It has never compared their meaning.