The training layer: a meta-aggregator / Dolma
Dolma
https://huggingface.co/datasets/allenai/dolma
infrastructureThe corporaactive
The archive's reading. A corpus that states its own sources and releases its curation toolkit: it settles, at the scale of the corpus, which kinds of writing enter the layer ('public-domain books', 'web content', 'scientific papers'), and makes the construction itself inspectable. What a model makes from a given work, and whether the work is carried with 'the metadata that lets it be found and attributed' (#186), are outside the saved abstract; the dataset card that would show document fields was rate-limited.
| Relation to this census | infrastructure |
|---|---|
| Record type | dataset |
| The question it answers | training membership by source type: which kinds of writing does a documented pretraining corpus hold, and how was it built? |
| Kind | an English corpus for language model pretraining research, released with its documentation and an open-source data curation toolkit |
| Operator | the paper's 36 authors, Luca Soldaini and 35 others (arXiv:2402.00159); the dataset card sits under the allenai namespace on Hugging Face, and its text was rate-limited (HTTP 429) and is unread, so the saved sources name no institution |
| Evidence | dataset; dataset documentation; research paper |
| Unit of record | not recorded in the saved abstract; the corpus is measured in tokens |
| What it records | a three-trillion-token English corpus built from web content, scientific papers, code, public-domain books, social media and encyclopedic materials |
| In the layer | the corpus is itself pretraining data: web content, scientific papers, code, public-domain books, social media and encyclopedic materials, released for research on language model pretraining |
| Under whose name | not recorded; the saved abstract does not say whether a document in Dolma keeps its author's name, source or attribution metadata |
| Found and read back | the paper states that Dolma is extensively documented (design principles, construction, a summary of contents) and that the curation toolkit is open-sourced for reproduction; a search or membership test for a single work is not recorded in the saved sources |
| What the apparatus makes from it | not recorded in the saved abstract beyond the corpus's purpose, research on language model pretraining; the paper reports analyses and experiments on intermediate states of the corpus |
| Method | curation of a mixture of sources into a pretraining corpus, with analyses and experiments on intermediate states of the corpus; the curation toolkit is open-sourced |
| Coverage | geography: English-language text (the abstract calls it an English corpus); period: not recorded |
| Count | value: three trillion tokens; as_of: arXiv v2, 6 June 2024 |
| Holdings | a three-trillion-token English corpus; sources: web content, scientific papers, code, public-domain books, social media, encyclopedic materials; documentation of design principles, construction and contents; an open-source data curation toolkit |
| Stated limits | the saved abstract states no limits of the corpus; it frames the release against a field where 'commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them' |
| Data access | the dataset on Hugging Face (allenai/dolma) and the code, both linked from the arXiv comments as 'this https URL'; the card itself was rate-limited and unread |
| Licence | not recorded in the saved sources (the arXiv page links a 'view license' for the paper; the dataset card, unread, would carry the dataset's licence) |
| Lifecycle | state: active; detail: released with the paper (arXiv v1 31 January 2024; v2 6 June 2024; accepted at ACL 2024); the dataset card was rate-limited on 2026-11-01, so later versions or the dataset's current state are unread |
| Version and time | paper_v1: 2024-01-31; paper_v2: 2024-06-06; venue: ACL 2024; read: 2026-11-01 |
| Founded | 2024 (arXiv v1, 31 January 2024) |
What its record can settle
- which kinds of source a model trained on Dolma has read (web, scientific papers, code, public-domain books, social media, encyclopedic materials)
- the corpus's size and language as its paper states them
- how the corpus was built, from its documentation and released toolkit
What it cannot settle
- whether a named work is in Dolma (no search or membership test is recorded in the saved sources)
- whether a work in Dolma keeps its author's name
- which models other than the authors' own were trained on it
Kept apart
- NOT a closed commercial corpus: it is released with documentation and its curation toolkit
- NOT a search tool for single works: the saved sources record no membership test
- NOT multilingual by its own description: 'a three-trillion-token English corpus'
- NOT a record of authorship: the saved abstract says nothing of attribution metadata
Evidence
- title (the resource's own statement):
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - authors (the resource's own statement):
by Luca Soldaini and 35 other authors
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - record_object (the resource's own statement):
we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials.
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - purpose (the resource's own statement):
To facilitate scientific research on language model pretraining
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - context (the resource's own statement):
commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them.
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - documentation (the resource's own statement):
We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents.
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - method (the resource's own statement):
We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices.
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - toolkit (the resource's own statement):
we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - venue (the resource's own statement):
Accepted at ACL 2024; Dataset: this https URL Code: this https URL
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - v1 (the resource's own statement):
[v1] Wed, 31 Jan 2024 20:29:50 UTC (7,715 KB)
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612… - v2 (the resource's own statement):
[v2] Thu, 6 Jun 2024 18:46:40 UTC (6,680 KB)
https://arxiv.org/abs/2402.00159 · read 2026-11-01 · http 200 · sha256 1203d77da94d4612…
Unchecked
the Hugging Face dataset card (dolma--card): HTTP 429, rate-limited, unread and unquoted; it would carry the dataset's operator, licence, versions, document fields and any per-document source metadata; the paper's full text (PDF and HTML) beyond the abstract: the unit of record, per-source counts and any attribution or metadata fields; the code repository linked from the arXiv comments
Cite: Lee Sharks, ed. "Dolma." The training layer: a meta-aggregator, traininglayerliterature.org, compiled 2026-11-01. https://www.traininglayerliterature.org/aggregators/dolma