The quote was verbatim and the citation was still wrong

We built a gate that makes a research agent quote a paper’s method and population word for word, so it cannot summarise a study it never opened. It works. It caught paraphrases, it caught a paywalled abstract dressed up as a read paper, and it forced two sources off the site entirely because no quotation could be produced.

Then a citation passed every check and was still wrong.

An agent came back with an ICSE paper on onboarding, reporting a significant negative correlation between onboarding and time-to-first-commit. That is exactly the outcome measure we wanted for a control about automating project setup — one command, clean checkout, working build. The abstract was verified. The direction was right. The venue was real.

Getting the PDF and reading it took ten minutes and produced this sentence, from the authors:

Such onboarding is more than giving a tutorial on creating a feature
branch or running a test suite.

The paper is about human mentoring. Teaching content, mentors explaining how the ecosystem relates to its individual projects, communication skills, community norms. Its numbers are excellent — median 45% lower time to first merged commit, bug-inducing commit probability 25% versus 14%, p=4.29e-77 — and none of them are about build automation. The one sentence that rules it out is in the discussion section, which an abstract-level fetch never reaches.

So the failure mode has two layers and the gate only catches the first.

Fabrication is an agent inventing content. A verbatim-quote requirement raises its cost enormously, because conclusions leak into every abstract and citation while a population — “1,281 (3x427) selected contributors” — exists in exactly one PDF.

Scope error is an agent quoting real content accurately and attaching it to the wrong claim. Nothing about quoting prevents this. The quote is true. The paper is real. The population is verbatim. And the citation asserts something the authors explicitly disclaim two sections later.

What makes it dangerous is that it is more plausible after the gate, not less. A record with a real DOI, a real venue, and two verbatim quotations reads as checked. It has been checked — for the thing the gate checks.

We have no mechanical fix. The gate we added is a rule that a source page must carry a does-not-show field, and that the field must name the scope, not just the confounds. It is a prompt for a human to notice, not a check. The only thing that actually caught this one was reading the paper, which cost ten minutes and one file on a desktop.

If your fleet does literature work: the abstract is where scope errors hide, because abstracts are written to travel and the caveats stay home.