Your idle detector is lying to you, and 38% of the time it is the same lie
If you run a fleet of coding agents headless, the single question you have all day is which one needs me right now. The usual answer is idle detection: watch each agent’s event stream, and when one goes quiet longer than some threshold, surface it. Mine used 45 seconds. It sorted quiet agents to the top of the board. It was the scheduling primitive the whole thing turned on.
It is wrong roughly a third of the time, and it is wrong in the direction that wastes your attention.
Here is the mechanism, and it is not a bug in anybody’s code. An agent’s transcript records a tool call when it starts and again when its result lands. Those are separate records written seconds or minutes apart. Any library tailing that transcript has to decide what to do about a call it has seen start and not finish, and the correct decision — the one the library I use documents carefully and implements well — is to withhold it. Advancing past an unresolved call would drop it permanently once the result arrives with nothing to pair against. So the stream stays silent from the moment a tool call starts until the moment it returns.
Which means a worker eight minutes into make test emits exactly nothing, and a worker waiting for a human emits exactly nothing, and they are the same row on your board.
I measured it across 400 worker sessions, 6,159 paired tool calls, 70.9 hours of wall time:
- median tool call: 0.4 seconds
- 7.6% of tool calls exceed a 45-second idle threshold — essentially all of them shell commands
- 38% of total worker wall time is spent inside a single tool call longer than 45 seconds
Thirty-eight percent. Not 38% of calls — 38% of the clock. For a third of the day, the board’s headline signal is reporting “needs you” about agents that are busy, and the false ones cluster precisely in the slices that run test suites, which are the ones you most want to leave alone.
Then it gets worse, in an interesting way. Five calls in that sample sat unresolved for between 30 and 390 minutes while carrying their own 20-minute timeout. The command cannot have run that long. The session was stalled — rate limited, suspended, something. And from the transcript, a stalled session and a running build are byte-identical: one record, no result. The signal that would separate them does not exist in the log at all. It is in the process table.
So I went to fetch better telemetry, which is where the second half of this comes in.
The agent harness turns out to emit real OpenTelemetry spans — a proper tree, one span per model call, one per tool, with the permission-wait broken out as its own child. The durations are measured, which the transcript-derived ones never were: reconstruct a duration from a transcript and you have silently folded the model’s thinking time into the tool’s execution time. Getting the real split was a revelation. One ordinary worker turn: six model calls totalling 23 seconds against six tool calls totalling 6.5. Three-quarters of that agent’s wall clock was spent thinking, not doing. I had been staring at that fleet for days with no way to know that.
And then I ran a 75-second tool call with a 5-second export interval, and watched the collector receive nothing at all for the entire 75 seconds. No span, no log, no metric. Spans export when they end. The new telemetry has exactly the same blind spot as the old, for a completely different reason.
That is the thing worth carrying away. Both systems are excellent at telling you what an agent did. Neither can tell you what it is doing. Those are different products, and if you are building fleet observability you will build the first one by accident while believing you are building the second, because the first one is what every library and every backend is shaped for. The live signal — is this process working or wedged — does not come from the telemetry pipeline at all. It comes from the unresolved tool call at the tail of the log, or from asking the operating system whether the child process still exists. Both are unglamorous, and both are about ten lines.
One last thing, free of charge, because it cost me two hours. I instrumented the fleet by adding environment variables to the workers’ shared settings file. It worked — one worker launched with it and exported a beautiful trace. Then it stopped, silently, and I spent a while investigating the collector. The settings file was generated. The supervisor loop rewrote it from a template every tick. It was tracked in git, sat among hand-maintained files, showed up in git status like anything else, and said nothing anywhere about being generated.
If configuration you edited stops taking effect and nothing errored, do not debug the consumer. Ask what writes the file.