The training layer: a meta-aggregator / OLMoTrace
OLMoTrace
https://allenai.org/blog/olmotrace
infrastructureInspectionReception becoming compositionactive
The archive's reading. An inspection instrument for #1543's FROM direction at the string level: a highlighted span is a witness that a stretch of output has a verbatim antecedent in the training data, the evidence for #1543's source status S = retrieved. Verbatim matching is not causal influence, and productive transformation (W, the generative uptake #1543 scores) leaves no verbatim span to highlight; whether a matched work is carried under its author's name is unrecorded.
| Relation to this census | infrastructure |
|---|---|
| Record type | tool |
| The question it answers | Which spans of a model's output appear verbatim in its full training data, and in which documents (training membership of output spans)? |
| Kind | a feature in the Ai2 Playground that traces language model outputs back to their full training data in real time; its paper calls it a system, released as open source |
| Operator | Ai2 (The Allen Institute for Artificial Intelligence), as a feature in the Ai2 Playground; the paper's authors are Jiacheng Liu and 30 others |
| Evidence | tracing tool; research paper |
| Unit of record | a span of a model response that appears verbatim at least once in the model's training data, shown with up to 10 document snippets from that data |
| What it records | verbatim matches between segments of a language model's output and documents in its training corpora, for Ai2's OLMo models (OLMo 2 32B Instruct, OLMo 2 13B Instruct, OLMoE 1B 7B Instruct) and for any model whose training data is accessible |
| In the layer | it matches output against the model's full training data: for OLMo 2 32B Instruct, five datasets across pretraining, mid-training, supervised finetuning, preference learning and RL, about 3.2 billion documents and 4.6 trillion tokens |
| Under whose name | not recorded; the side panel shows document snippets from the training data, and the saved sources do not say whether a document's author, title or source address is shown; Ai2 leaves it to the user to 'personally verify that these documents come from trusted sources' |
| Found and read back | yes, for verbatim spans: 'Show OLMoTrace' highlights long and unique spans of a response that appear verbatim in the training data and lists the documents that contain them (up to 10 snippets per span, a random 10 where there are more), ranked by BM25 against the prompt and response; a span may be covered only by parts found in different documents |
| What the apparatus makes from it | the model's response, of which only long, unique, self-contained spans with a verbatim match are highlighted (K = 0.05 times the response's token count); Ai2 shows it tracing 'creative' expressions, arithmetic and a false knowledge-cutoff claim to training documents, in its own hedged terms ('may have learned', 'could have easily learned it through memorization') |
| Method | infini-gram indexing of the training data and a parallel span-finding algorithm (O(L * log N)); spans meeting existence, self-containment and maximality criteria are ranked by span unigram probability, the lowest K kept, document snippets retrieved and ranked by BM25 |
| Coverage | geography: not recorded; period: the training data of the supported OLMo models; the 13B model's pretraining data has a cutoff of December 2022 |
| Count | value: about 3.2 billion documents and 4.6 trillion tokens matched for OLMo 2 32B Instruct; as_of: Ai2 blog post, 9 April 2025 |
| Holdings | tracing for three flagship models: OLMo 2 32B Instruct, OLMo 2 13B Instruct, OLMoE 1B 7B Instruct; matching against about 3.2 billion documents and 4.6 trillion tokens (OLMo 2 32B Instruct); results within a few seconds; source code released; paper at ACL 2025 demo track |
| Stated limits | 'The interpretation of text matches with training data is quite nuanced.'; generic spans with less relevant documents are shown in a less saturated colour and labelled 'low relevance'; 'We are still in an early stage in exploring the full potential of this tool'; it applies only to a model 'for which you have access to the training data' |
| Data access | public, in the Ai2 Playground ('Show OLMoTrace' on a model response); source code and tech report linked from the blog post; the paper on arXiv |
| Licence | not recorded; the paper states OLMoTrace 'is publicly available and fully open-source', and the saved sources name no licence |
| Lifecycle | state: active; detail: introduced 9 April 2025 as available in the Ai2 Playground; paper v1 9 April 2025, v2 8 July 2025; its availability after that date is unread |
| Version and time | blog: 2025-04-09; paper_v1: 2025-04-09; paper_v2: 2025-07-08; venue: ACL 2025 demo track; read: 2026-11-01 |
| Founded | 2025 (introduced 9 April 2025) |
What its record can settle
- which long, unique spans of a given OLMo response appear verbatim in its training data
- which training documents (up to 10 snippets per span) contain a highlighted span
- whether a phrase of a work held in the OLMo training data recurs verbatim in an output
What it cannot settle
- whether a matched document caused the output (Ai2's own wording is 'may have learned')
- exhaustive source attribution: at most 10 documents per span are shown, sampled at random beyond that
- anything about outputs with no long verbatim span, including paraphrase and transformed writing
- under whose name a matched document sits in the training data
- models whose training data is closed
Kept apart
- NOT a record of causal influence: Ai2's own wording is 'may have learned'
- NOT exhaustive source attribution: at most 10 documents per span are shown
- NOT a detector of paraphrase or transformation: it finds spans that appear verbatim
- NOT applicable to closed models: it needs access to the training data
- NOT a corpus: it reads the OLMo training datasets and holds none of its own
Relations
- neighbour: Data Portraits (both let a text be checked against a model's training data from outside the model's weights: Data Portraits records the corpus for membership queries, OLMoTrace matches output spans verbatim against the full corpus)
Evidence
- self_description (the resource's own statement):
Today we introduce OLMoTrace, a one-of-a-kind feature in the Ai2 Playground that lets you trace the outputs of language models back to their full, multi-trillion-token training data in real time.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - date (the resource's own statement):
April 9, 2025
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - authors_blog (the resource's own statement):
Jiacheng Liu et al. - Ai2
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - operator (the resource's own statement):
© The Allen Institute for Artificial Intelligence - All Rights Reserved.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - purpose (the resource's own statement):
We developed OLMoTrace to enable researchers, developers, and the general public to inspect where and how language models may have learned to generate certain word sequences.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - models (the resource's own statement):
OLMoTrace is available today with our three flagship models: OLMo 2 32B Instruct , OLMo 2 13B Instruct , and OLMoE 1B 7B Instruct , and it can be applied to any language model for which you have access to the training data.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - unit (the resource's own statement):
The highlights indicate long and unique spans that appear verbatim at least once in the training data of this model.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - partial_cover (the resource's own statement):
Sometimes, a highlighted span may not be present contiguously in any single document, but different parts of that span are present (possibly in different documents) and together they cover the entire span.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - limit_nuanced (the resource's own statement):
The interpretation of text matches with training data is quite nuanced.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - name_verify (the resource's own statement):
If you personally verify that these documents come from trusted sources, you can gain confidence that this piece of information generated by the model is factually correct.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - low_relevance (the resource's own statement):
We use a less saturated color to highlight spans with less relevant documents, and they are labeled in the document list as “low relevance.”
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - creative (the resource's own statement):
In creative writing tasks there is less of a need for fact checking, but you can still trace certain LM-generated expressions back to the training documents.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - memorization (the resource's own statement):
OLMoTrace reveals that this entire expression appears multiple times in the training data, so the model could have easily learned it through memorization.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - cutoff_case (the resource's own statement):
now we can see that the model may have learned to say August 2023 from the post-training examples listed on the right.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - period (the resource's own statement):
since its pre-training data has a cutoff of December 2022
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - limit_early (the resource's own statement):
We are still in an early stage in exploring the full potential of this tool
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - criteria (the resource's own statement):
Maximality : The span is not a subspan of another span that meets the above two criteria.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - method_index (the resource's own statement):
To speed up computation, we index the training data with infini-gram and develop a novel parallel algorithm, reducing the time complexity of span finding from the naive O(L^2 * N) to O(L * log N)
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - method_k (the resource's own statement):
keeps the K spans with the smallest unigram probability, where K = 0.05 * the number of tokens in the model output.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - snippets (the resource's own statement):
For each kept span, OLMoTrace retrieves up to 10 document snippets enclosing this span from the training data, which are displayed in the side panel. If there are more than 10 documents for a span, a random set of 10 is sampled.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - bm25 (the resource's own statement):
in the document panel we rank all documents by a BM25 score in descending order.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - count (the resource's own statement):
In total, these five datasets contain about 3.2 billion documents and 4.6 trillion tokens, and OLMoTrace matches against all of them.
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - code (the resource's own statement):
We also release our source code .
https://allenai.org/blog/olmotrace · read 2026-11-01 · http 200 · sha256 39e59dda3ba78e0b… - paper_abstract (the resource's own statement):
OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora.
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - speed (the resource's own statement):
our system returns tracing results within a few seconds.
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - creativity (the resource's own statement):
We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models.
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - license (the resource's own statement):
OLMoTrace is publicly available and fully open-source.
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - authors (the resource's own statement):
by Jiacheng Liu and 30 other authors
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - venue (the resource's own statement):
ACL 2025 demo track
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - v1 (the resource's own statement):
[v1] Wed, 9 Apr 2025 17:59:35 UTC (6,674 KB)
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4… - v2 (the resource's own statement):
[v2] Tue, 8 Jul 2025 00:56:11 UTC (6,836 KB)
https://arxiv.org/abs/2504.07096 · read 2026-11-01 · http 200 · sha256 dfdb383596de5bf4…
Unchecked
the tech report's full text and the source code repository (linked and unsaved); the Ai2 Playground itself: whether the feature is still offered, and whether the document panel shows a document's source address or author; the five training datasets named in the post (olmo-mix-1124, dolmino-mix-1124 and the post-training mixtures): the saved sources do not say how they relate to the Dolma corpus row, so no relation is recorded
Cite: Lee Sharks, ed. "OLMoTrace." The training layer: a meta-aggregator, traininglayerliterature.org, compiled 2026-11-01. https://www.traininglayerliterature.org/aggregators/olmotrace