Docs
AdvocacyLab
Tools for people who write evidence-based reports, such as Country of Origin Information and human rights research.
They find sources, collect quotations while a report is written, check every quotation against its source before publication, and read a draft as its critics will.
Built by Data Activists, a nonprofit that works with refugee charities on research, and on data and AI.
- 89%
- of quotations automatically and verifiably settled by CiteCheck
- 0 of 30,000+
- CiteFind passages that differ from their source
- 63 of 82
- arguments in real published replies that ReportReview predicted
A COI report is trusted because each claim can be followed to its source, and one misquoted figure can undermine it on appeal. Researchers already use AI to find sources and to write, as the figures below show, and chat tools write fluent text with plausible citations, but a quotation that comes from AI is no evidence of what a source says. Checking every quotation by hand doesn't scale: in the reports we have run, the median one holds about 35 quotations and a tenth hold over 170. At a few minutes each to open the source and find the words, that is nearly two hours for a typical report and more than a working day for a large one, so reviewers check samples.
AI is already in published COI
From our scans of ecoi.net and ReliefWeb, in an upcoming research paper.
- 1 in 6
- reports on ecoi.net since 2025 with 100+ links cite links copied from ChatGPT. Publishers include national asylum authorities and the EUAA.
- 1 in 30
- of all reports since 2025 cite a link copied from ChatGPT, both on ecoi.net and in our sample of ReliefWeb. For some publishers, it's 1 in 4.
- 90%
- of the humanitarian publishers we compared on ReliefWeb use words typical of ChatGPT more since 2023.
- +75%
- is how much more often the typical publisher among them uses those words since 2023.
The tools
One for each stage of a report. Pick one to see it.
One rule runs through every tool
A model may point at text, but only code extracts it.
Every passage on screen was extracted from a real document by code, and a person makes every decision. So everything the tools show can be traced back to its source.
Rules a model can't override
No model writes a quotation, a passage, or an address
- Model gives a passage's first and last words
- Code finds them in the real text
- Code cuts the text between, or drops the passage
Of more than 30,000 CiteFind passages, none differed from its source. Addresses come only from the report's own links or from search results.
Code decides whether a quotation is in its source
- word for word
- ignoring punctuation
- small differences
The table says which. A changed figure never matches:
Models only read what code couldn't confirm: whether a quotation is a paraphrase or a translation, or whether the words that differ change its meaning. Their answer never turns a row green, and anything code hasn't confirmed is flagged for a person to review.
- An unreadable source is never called missing. Red means only that no passage in the source resembles the quotation. A captcha, a login or a dead link is a source issue for a person.
- No summaries, because a summary is where a model starts making claims. ReportReview's points are labelled as arguments to prepare for.
What we measured Twelve figures, each against an answer the tool couldn't see.
Each figure is measured against an answer the tool couldn't see: reviewers' own answers, real published replies, or the facts in published query responses.
| Tool | Measure | Result |
|---|---|---|
| CiteCheck | Quotations reviewers confirmed missing that it marked found | 0 of 40 |
| Quotations settled without a person, on reports known to be accurate | 89% | |
| Quotations found in their source, across every report run | 70% of 3,762 | |
| Quotations whose source couldn't be read, dead links aside | 4% | |
| Quotations in a COI charity's reports that it picked out | 100% | |
| CiteFind | Passages that differ from their source | 0 of over 30,000 |
| Judged passages that were useful | 138 of 150 | |
| Judged passages that were off topic | 0 of 150 | |
| A published query response's facts found in what it returned | 49 of 136 | |
| ReportReview | Arguments in real published replies that it predicted | 63 of 82 |
| The same for Claude Opus, given the report alone | 62 of 82 | |
| Planted errors found, in a report kept back from tuning | 10 of 10 | |
| Points read by hand that were wrong | 0 of 24 |
CiteCheck: 74 NGO reports (3,762 quotations, 824 sources) checked from the live server, reviewers' answers on 217 rows, and 12 ecoi.net and ReliefWeb reports checked by hand. CiteFind: 45 query responses chosen at random from ecoi.net, 15 for tuning and 30 kept back. ReportReview: 8 published Amnesty International reports whose targets replied, scored by a model that agreed with hand scoring 89.6% of the time, and a ninth seeded with 10 errors. Where a run was repeated, the table gives the best. Repeated runs vary by 5 to 9 arguments in ReportReview and about 7 facts in CiteFind, so smaller differences are noise. CiteCollect uses CiteCheck's check; its own use will be measured on its first live reports.
What happens to your text Open models on providers that keep nothing, and everything deleted on a date.
- Open models, on providers that keep nothing. The tools run on open models (DeepSeek, GLM and Mistral), which on these tasks do as well as the large commercial models or better: in ReportReview's test they predicted 63 of 82 arguments, against 62 for Claude Opus. They are served by zero-retention providers in the EU, UK and US, which neither keep the text nor train on it. For an organisation that requires it, we can limit that to the EU and UK.
- Only addresses and search queries reach the web, never a report or CiteFind's question. With 10 made-up case questions, no query kept a name, number or birth year, and the page still warns against putting case details in a question. The web search services don't train on queries.
- Everything is deleted on a date. An uploaded report and its review go after 30 days, a CiteFind search after 30, and a CiteCollect report's case text 90 days after it is finished or 90 after it was left idle. Anything can be deleted sooner.
- Nothing that identifies a report is stored or mailed. Files are stored under opaque names, pictures of a report's pages are never stored, and an email says only that work is ready.
- AI can be switched off for a whole workspace, for a policy that needs AI use approved. CiteCollect's check by code still runs.
- Or run it yourselves, on your own servers with your own model keys.
Limits What a red row means, and what each tool is for.
- A red row isn't proof. The words aren't in the text the tool read, which may be a page about the document. The row says to open the link before striking anything.
- Some sources stay out of reach: captchas, social media logins and well-hidden PDFs. Those rows are left to a person.
- A paraphrase verdict is one model's opinion, and a quotation found word for word can still come from a weak source.
- CiteFind is a starting point for research, not the whole search. It brings back usable passages in about 80 seconds: of 150 judged, 138 were useful and none off topic. A researcher still searches further before relying on it.
- ReportReview prepares a team for the reply; it doesn't replace a reviewer. It anticipated about three-quarters of the arguments in real published replies (more than Claude Opus given the same report).
- CiteCollect is the newest tool, and checks quotations with CiteCheck's check. Most figures come from English-language reports.
AdvocacyLab is made by Data Activists C.I.C., a UK community interest company (no. 17082883).