Research
CiteFind Beta
CiteFind turns a research question into passages from real sources, each cut word for word from the source's own text and shown with its address.
How it works
- 1
A researcher types a question, for example on the treatment of returnees in a given country.
- 2
A model writes a few search queries from it. Only the queries go to the COI indexes (ecoi.net, ReliefWeb) and a web search. The question itself is never sent to a search service.
- 3
CiteFind reads 20 of the sources that come back, about 8 from the COI indexes and 12 from the web, and shows the passages that answer the question, ready to add to a CiteCollect report.
- 4
A search takes about 80 seconds. If the researcher has left the page, they get an email when it is ready.
It cannot invent a quote or a source, by design. The model never writes a passage or an address. It only points at where a passage starts and ends, and code cuts that passage from the source's fetched text. If the model points at something that is not in the text, the passage is dropped. Every source shown was fetched and read, and its address comes from the search service, not from a model. CiteFind writes no summaries, because a summary is where a model would start making claims.
Working through a search
-
Each source is a numbered card with its passages inside it. Cards fold, and can be ordered by relevance or newest first.
-
The first screen shows 30 passages, one for each claim and at most 4 from any one source. Passages from other sources making the same claim are grouped under it, and the rest are under More passages.
-
Each source is labelled primary, secondary or tertiary. The label only changes the order. A tertiary source is still read, and still shown.
-
A claim can be marked less relevant or moved up, and the first screen stays as the researcher arranged it.
-
A passage or a whole source can be removed. Removing deletes it, so the CSV download matches the page.
-
A passage goes into a CiteCollect report in one click, as a draft citing the document it was cut from.
-
Every account starts with an example search, so a researcher can see the result before running their own.
What we measured
No passage has ever differed from its source: 0 of more than 30,000, across every test run.
0
Passages of more than 30,000 that differ from the source text
0 of 150
Judged passages off topic; 138 were useful
~80 s
Per search, at most 2 minutes
| Measure | Result | Checked by |
|---|---|---|
| Passages that differ from the source text | 0 of more than 30,000 | Code, every passage |
| Passages on topic | 138 of 150 useful, 12 marginal, 0 off topic | A model, with a second model as check |
| The report's facts found anywhere in the results | About a third of its quotations (49 of 136 on the newest held-out set) | A model |
| The same, on the first screen only | 31 of 136 | A model |
| Control: the same quotations against another country's results | 1 of 15 (a judging error) | A model |
| Strict test: the same fact, for the same actor, place and period | 40 to 41 of 251 quotations anywhere in the results, 28 to 36 on the first screen | Two models, which must both agree |
| Control for the strict test | 0 or 1 of 30 | Two models |
| The report's own cited addresses found | About 1% | Code |
| Primary sources | 217 of 299 sources (79 secondary) | A model |
| Sources read | 266 of 300 | Code |
| Time per search | About 80 seconds, at most 2 minutes | Code |
The control is why the fact counts can be trusted: checked against another country's results, the same quotations almost never match. The same search run twice moves the strict count by about 7, so smaller differences are noise.
Where the gap is
To tell finding from reading, we gave CiteFind each report's own cited sources in place of a search. Its passages then stated 129 to 141 of the 251 quotations, against 40 to 41 from its own search. Only 170 of the 251 are in any cited source we could fetch at all; the rest sit behind paywalls, at dead links, or in files that can't be read. So CiteFind reads the right sources well, and the gap is in finding them.
Of 251 quotations from 30 reports, in the strict test.
That gap may be partly the measure's. Another good source rarely states a report's exact fact for the same actor, place and period, and the web has moved since the reports were written. We have not yet measured how far two human reports on the same question overlap, or whether CiteFind's sources are better or worse than a report's own.
Tried and left out
- Searching the question in parts. A model split it into up to eight parts, each searched apart: about 10 more facts anywhere in the results, none on the first screen, at more than twice the cost.
- More web search requests a search (three instead of one): about 12 more facts anywhere in the results, none on the first screen. Kept, switched off, in case the passages below the first screen start to matter.
- Reading 30 sources instead of 20 found nothing more and took a minute longer.
- Another search service (Exa, ranked first on a public benchmark) returned 1.6 to 1.9 times as many of the reports' own addresses for the same queries, but once the whole search ran it was within 2 facts of the current one.
- Searching only for sources older than the report found nothing more, so the measure is not penalising newer sources.
- A model asked to list every relevant passage instead of the best six: 10 more facts anywhere in the results, 10 fewer on the first screen, because the ranking can't tell twice as many passages apart.
- Following the links printed in the COI reports CiteFind found: 11 more facts, the largest gain of any change, but it read nearly three times as many sources and most of the gain came from 3 or 4 reports.
The test set
We tested on 45 COI query responses published on ecoi.net. A query response is a researcher's question with the sources they found to answer it, so each one comes with its own answer to compare against.
- Publishers
- The EUAA (EU Agency for Asylum), 24; Ireland's Refugee Documentation Centre, 21.
- Countries
- Afghanistan 7, DR Congo 6, Iran, Iraq, Palestine, Russia, and Syria 5 each, Sudan 4, Lebanon 2, Ukraine 1.
- Dates
- 2023 to 2026, most from 2024.
- Chosen at random
- From 130 English-language responses published since 2023. Each question is the response's own, unedited title.
- Held out
- 15 were used to tune CiteFind and 30 were kept back to test it. The fact, source, and time figures come from those 30; the on-topic sample is from the tuning 15 and the verbatim count covers every run.
- The strict test
- 10 quotations from each of 30 reports (15 tuning, 15 held out), 251 in all.
What it is not
Not a bibliography rebuilder
It rarely finds the exact sources a given report cited: about 1% of their addresses come back. It finds other sources stating some of the same facts.
Not the same twice
The same question run again shares about 40% of its sources.
Not a judge of sources
A word-for-word passage can still come from an old, mistaken or biased source. The researcher decides what to use.
Not a complete search
The page says so, and a researcher should still search the usual places.
Not yet measured on the live server
Every figure above was measured from a laptop. Some sites answer the live server differently, so its share of sources read may differ.
What happens to the question
- The question goes only to the models that write the queries and pick out passages. Their providers hold zero-retention agreements: they don't keep it or train on it.
- Only the queries go to the COI indexes and the web search service, which doesn't train on them. In a test with 10 made-up case questions, the queries kept no name, number or birth year, but the page still warns against putting case details in a question.
- After 30 days the whole search is deleted automatically: the question, the queries, the sources and the passages. Only counts are kept.
- The email saying a search is ready carries two counts and a link, never the question.