Before publication
ReportReview
ReportReview reads a draft report before it is published and tells the team two things: what the government or company the report criticises is likely to say in its reply, and where the report disagrees with itself. Each point is pinned to the passage it is about, on the report's own pages, so the team can fix the draft before the other side gets the chance to answer it.
How it works
- 1
A researcher uploads the draft (PDF or Word) and names who will answer it.
- 2
Three readers go through the report separately. Two write the replies the other side is likely to make; the third checks the report against itself, for figures, dates and claims that don't agree.
- 3
A model writes a few search queries from the report's names and terms, and a web search finds what the other side has already said in public. The readers see those pages as numbered notes.
- 4
A writer joins the three readings into one list of points. It never sees the report, so it can only combine what the readers said.
- 5
Each point is placed on the passage it is about. A whole review takes about 7 to 9 minutes; while it runs, the page shows which readers have finished, and if the researcher has left the page they get an email when it is ready.
Working through a review
-
The review opens beside the report's own pages, with each point's passage highlighted. Points can be read strongest first or in page order, and stepped through with Previous and Next.
-
Each point is marked done or deleted, and the list filters to what is still to do. Deleting asks why in one tap: wrong, too general, or already in the report. Those counts tell us which reader to improve.
-
A finished review downloads as Markdown, and a review can be deleted from the list at any time.
-
Every account starts with a finished example: a review of Amnesty International's published report on Libya, shown beside its pages, so a team can see what a review looks like before uploading a draft of its own.
What it cannot make up
It cannot put words in the report's mouth or invent a source, by design.
The rules are in code, not in a prompt, so no model can talk its way past them.
-
Passages from the report are cut, never typed. A model only gives the first and last words of the passage a point is about; code finds them in the report and cuts the passage out. If those words aren't in the report, the point is shown with no passage rather than a guessed one. A point about a contradiction can rest on two passages, one for each side of it.
-
Every source is a page the search actually returned. A reader can only cite a note by its number, and code turns the number into that page's address. A number that matches no note is removed from the point. No model ever writes an address.
-
A cited page has to back the point. A second model checks each point against the page it cites, and removes the citation where the page doesn't say it. The point itself stays.
-
A quotation of the report that isn't in it is flagged. When a consistency point puts words in quotation marks, code checks them against the report and marks any it can't find on the page.
The proposed replies are a model imagining the other side, and are presented that way: they are arguments to prepare for, not facts about the report.
What we measured
We ran ReportReview on published reports whose targets had published their replies, so each report comes with the answer it actually got. Across 8 reports, those replies made 82 distinct arguments.
63 of 82
Arguments from the real replies that the review predicted
10 of 10
Errors planted in a report no prompt was written from
0 of 24
Points read by hand that were wrong
| Measure | Result | Checked by |
|---|---|---|
| Arguments from the real replies that the review predicted | 63 of 82 (53 firmly) | A model, which agreed with hand scoring 89.6% of the time |
| Of those, arguments the report had not already answered | 34 of 48 | The same |
| The same review without the web search | 57 of 82 (44 firmly) | The same |
| The web search, compared on identical readings with and without it | More arguments in 5 of 5 comparisons: 4 to 9 more, 4 to 12 more firmly | The same |
| A leading general model (Claude Opus) given the report alone | 62 of 82 | The same |
| Errors planted in a report no prompt was written from | 10 of 10 (8 firmly without the consistency reader) | By hand |
| Sample of 24 points read by hand | 0 wrong; 8 checkably right, 13 plausible but not checkable without outside facts, 3 too vague or partial | By hand |
| Points placed on a passage of the report | 89%; all 14 placed points read by hand were on a relevant passage | Code, then by hand |
| Citations the source check removed | 8 of 315, 7 of them rightly | By hand |
| Time per review | About 7 to 9 minutes | Code |
Where a run was repeated, the table gives the best of each figure. The same review run twice moves by 5 to 9 arguments of 82, so smaller differences are noise.
Tried and left out
- A faster setting cut a review to about 3 minutes and predicted as many arguments, but named fewer of them firmly in all three runs (46 to 50 against 50 to 53). We kept the slower one.
- A deeper research search, which reads hundreds of pages instead of dozens, predicted no more arguments than the standard search and added 3 to 12 minutes.
- A second model checking each point caught none of the 3 wrong points it was shown.
- Other models in each reader's seat (GLM in place of Flash, Kimi K3.1 anywhere) predicted no more arguments, at more cost.
The test set
- Reports
- 8 published Amnesty International reports (one with the Omega Research Foundation), each criticising named companies, among them Shell and Eni, Google and Facebook, Airbnb and Booking.com, and JBS.
- Answer key
- The companies' own published replies, split into 82 separate arguments.
- Planted errors
- A ninth report (Amnesty, on Libya) seeded with 10 errors: figures that don't add up or change between the summary and the body, a person's age given two ways, a figure with no source, a claim cited to a document older than it, and a recommendation about something the report never discusses. It was kept back until the design was fixed.
What it is not
Not a substitute for a right of reply
It predicts the arguments a reply is likely to make; the other side may still raise something it missed, as it did with about a quarter of the arguments in the test set's real replies.
Not tested on government targets yet
Every test report criticised companies. Reports aimed at a government or its agencies, which is most COI and asylum work, are the next thing to measure.
Not always precise
Points are broader than a top model's would be: in the hand-read sample a top model gave 17 of 24 checkably right points, ReportReview 8. Neither gave a wrong one. Some points also repeat each other between the reply list and the consistency list.
Not a fact-checker of the outside world
It checks the report against itself and against what the other side has said in public. Whether an outside claim is true is still the researcher's call. The source check removes citations that plainly don't fit; the ones it keeps have not yet been read by hand.
Weaker on two-column layouts
Two-column PDFs can reach the readers with their columns interleaved, which blurs a review without any sign on the page.
What happens to the report's text
- The report goes to the model providers under zero-retention agreements: they don't keep it or train on it.
- Only search queries go to the web search service, which processes them only to answer and doesn't train on them. The report itself is never sent there.
- After 30 days the file, its text and the review are deleted automatically: the readings, the points and their passages. A team can delete them sooner from its list. Only counts are kept.
- The pictures of the report's pages are drawn when the review is opened and never stored.
- The email saying a review is ready carries no title, file name or words from the report.