Ask the agent for the population, not the conclusion

Seven research agents came back with 41 sources. Every record had a plausible title, a real venue, and a confident one-line finding. The field that caught the bad ones was not the finding.

The output schema required method and population as verbatim quotations from the source. Not summaries. Quotes, with the quotation marks in the string.

That is the whole gate, and it works because of an asymmetry. An agent that could not open a paper will still produce a confident conclusion — conclusions are what a literature is about, they leak into every abstract, citation and blog post, and reconstructing one from memory is nearly free. A population is not. 357 real faults in 5 open-source applications that comprise a total of 321,000 lines of code is a specific string that exists in one PDF. Inventing it convincingly means inventing a study.

Behaviour split three ways under that constraint. Agents that hit 403s wrote "method": "PENDING: paywalled, abstract only" and said which route they had tried. One substituted the authors’ own conference slide deck for a blocked paper and disclosed the substitution inside the record rather than passing it off. One returned a status line where the data should have been, and had to be sent back.

One source did not survive. A well-known four-team industrial case study, defect density down 40–90%. No verbatim population, so it is not published. The control it would have supported now has counterevidence and nothing on the other side.

Then the wrong turn, which was mine. The check tested whether the parsed value began and ended with a quote character. YAML strips outer quotes, so population: "We analyzed months of..." parses to a string with no quotes in it at all, and four of my own sources failed the gate I had just written. The fix is json-encoding a value that already contains literal quote characters: population: "\"We analyzed months of...\"".

A fabrication gate holds when the cheapest fabrication costs more than the retrieval. Conclusions are cheap. Ask for the number nobody remembers.