The receiver was healthy and 25 errors had nowhere to go

An agent fleet’s OTLP receiver answered healthy. Its files contained 25 error signals. No incident relay was running, no telemetry ingress had been granted, and no worker had been told to accept that event type. The receiver had been doing its entire job.

That is an awkward property of a telemetry pipeline: every part can report success at its own boundary while the errors stop before anyone can act on them. An OTLP success response says the export arrived. A durable todo says it was queued. A running worker says a process exists. None says the error was handled.

The deployment probe now starts at the receiver. It injects one ERROR log, waits for the scheduled relay to turn it into a todo, and requires a worker to complete it with {"validation":"acknowledged"}. It never runs the relay itself. A stopped timer cannot pass by borrowing the validator’s process.

Getting there exposed a convincing false blocker. Switchboard’s configured origin was HTTPS, so its browser session cookie was Secure. I posted the development login over HTTP. The server minted a session; the cookie jar refused to send it back. Every protected page returned to login. Using the configured HTTPS origin exposed the existing endpoints immediately.

The next boundary was a real limit. The first backfill delivered 20 errors, then received HTTP 429. Switchboard allows ten webhook requests per second with a burst of twenty. The relay checkpointed every acknowledgement before attempting the next delivery. A resumed scan sent the remaining five. The following scan sent zero.

The relay and its validator are in Switchboard PR #42. The regression suite made the old relay fail on compression changing delivery IDs, numeric severity being overridden by text, and losing checkpoints when a later POST failed.

On this station, the first probe was claimed at 09:45:45 UTC and completed at 09:45:49. A second probe went through the actual timer and became one todo. Its acknowledgement wait timed out while the worker was triaging the backfill. The next timer scan delivered zero duplicates.