Before publication
CiteCheck
CiteCheck checks a finished report's quotations against their sources. It reads every quotation and its cited address out of the report, fetches each source, and says for each one whether the words are actually there, so a reviewer only has to look at the ones that aren't.
How it works
- 1
A reviewer uploads the report (PDF or Word).
- 2
A model points at the quotations and says which of the report's own links each one cites. Code cuts the quotation from the report and takes the address from the file.
- 3
CiteCheck fetches each source. Where a page is only a landing page, it reads the PDF the page links to; where a site blocks bots, it asks a service that retrieves web pages; where a source is gone, it tries the web archive.
- 4
Code looks for each quotation in the source's text: word for word, then ignoring punctuation, then allowing small differences. The table shows which, and highlights the passage it found.
- 5
Where the words aren't there, a model reads the closest passages and says whether the quotation is a paraphrase, a translation or absent. That reading sits beside the result and never changes it.
Most reports take about 45 seconds. A large one can take several minutes, and an email says when it is done.
What it cannot make up
It cannot invent a quotation, an address or a verdict, by design.
The rules are not AI-dependent: they're a Ctrl-F with some custom code, so a model has nothing to make up.
-
Every quotation and every link come from the report itself. A model only points; code cuts the words from the report's text. An address a model gives that isn't one of the report's own links is thrown away, and the cell is left empty.
-
Whether a quote is in its source is decided by code. The same quote against the same source always gives the same answer. A model's reading of a paraphrase is shown next to that answer, labelled as a reading, and is never the verdict.
-
Red means only one thing: no passage in the source resembles the quote. A page that wouldn't load, a captcha or a login wall is shown as a source issue, never as a missing quote.
-
A changed figure never counts as a match. "47 cases" against "147 cases" is flagged, though the strings are 99% alike, because an altered number is the commonest fabrication.
-
Small wording changes are shown, not smoothed over. Where the words differ, the table shows which words, and a model says whether the change alters the claim (a dropped "not" is still 99% alike).
What we measured
On reports we know to be accurate, CiteCheck settles 89% of quotations by itself and leaves the rest for a person to check.
0 of 40
Quotations reviewers confirmed missing that the tool turned green
89%
Quotations settled by the tool on accurate reports
~45 s
For most reports; a few minutes for the largest
| Measure | Result | Checked by |
|---|---|---|
| Quotations settled on accurate reports | 89% | Code |
| Quotations found in their cited source, in full or in part, across every report run | 70% of 3,762 | Code |
| Quotations whose source wouldn't load or couldn't be read, dead links aside | 4% | Code |
| Quotations reviewers confirmed missing that the tool turned green | 0 of 40 | Reviewers, by hand |
| Quotations reviewers found that the tool now finds, or reads as a paraphrase or translation | 98 of 177 | Reviewers, by hand |
| Quotations in a COI charity's reports the tool picked out | 100% | By hand |
| Rows that aren't quotations (article titles, organisations describing themselves) | 1% | By hand |
| Time per report | About 45 seconds; a few minutes for the largest | Code |
The 0 of 40 is the number that matters most: a tool that waves through a missing quote is worse than none.
The test set
- Every report run on the platform
- 74 reports, 3,762 quotations and 824 sources, checked from the server CiteCheck runs on.
- Reviewers' own answers
- Where reviewers had marked a row themselves, their answer is the key: 40 quotations they confirmed missing, and 177 they found that the tool had not.
- A hand-checked set
- 12 reports from different NGOs and in different formats, found on ecoi.net and ReliefWeb, checked by hand for which quotations the tool picks out and which link it gives each one.
- Accurate reports
- Reports we know to have no missing quotations, which are closer to what a careful team publishes.
What it is not
Not proof that a citation is wrong
A red row means the words aren't in the text CiteCheck read. That text can be a page about the document rather than the document, so the row tells the reviewer to open the link before striking the citation.
Not able to reach every source
Captchas, social media logins and well-hidden PDF links still stop it. Those rows say the source couldn't be read and are left for a person, never marked as missing.
Not a judge of paraphrase
Where a quotation is reworded or translated, the model's reading is a second opinion from one model, with that model's blind spots. A person still decides.
Not a judge of sources
A quotation found word for word can still come from an old or unreliable source.
What happens to the report's text
- The report goes to the model providers under zero-retention agreements: they don't keep it or train on it.
- Only a quotation's cited address leaves the server to fetch the source. The report itself is never sent there.
- The uploaded file and its text are deleted 30 days after upload, and the file is stored under a name that says nothing about it.
- The email saying a check is done carries no title, file name or quotation.