The training layer: a meta-aggregator / The Pile
The Pile
infrastructureThe corporahistorical
The archive's reading. The archive reads the Pile as the training layer named and itemized: a work enters by belonging to a component (Books3, PG-19, Pile-CC), and what the receiving apparatus makes from it is a trained model, scored on the Pile's own benchmark. Its own pages offer no per-work search; the corpus has since been inspected from outside (Data Portraits documents it), and Wikipedia's record of the Books3 removal and the authors' suit is the field's clearest case of works found in the layer under their authors' names after the fact.
| Relation to this census | infrastructure |
|---|---|
| Record type | dataset |
| The question it answers | training membership by component: which 22 component datasets make up the training set, and which of them carries a given work? |
| Kind | open source language modelling data set: an English text corpus targeted at training large-scale language models |
| Operator | EleutherAI (the home page footer reads 'EleutherAI 2021'; the paper's authors are Leo Gao and 11 others) |
| Evidence | dataset; dataset documentation; research paper |
| Unit of record | a component dataset (one of 22) and, within it, a document; distributed as jsonlines compressed with zstandard |
| What it records | 22 diverse subsets, both existing and newly constructed, many from academic or professional sources; per Wikipedia, books, movie transcripts and scientific papers among other media, with components including Pile-CC, PubMed Central, Books3, OpenWebText2, arXiv, GitHub, Gutenberg (PG-19), OpenSubtitles, Wikipedia, YouTube Subtitles and Enron Emails |
| In the layer | the texts of its 22 components, assembled for training large language models; per Wikipedia, the Books3 component held copyrighted material compiled from Bibliotik, a piracy website, and was removed from the Pile; tens of thousands of YouTube videos had their subtitles scraped into it |
| Under whose name | not recorded in its own saved pages; per Wikipedia, three authors filed a class action in 2024 over the Books3 component |
| Found and read back | not recorded: its own pages offer download and a bits-per-byte benchmark (Pile BPB), with no per-work search; per Wikipedia, by 2024 it was taken down from its original site and remained accessible from other file sharing services |
| What the apparatus makes from it | language models trained on it: its own page scores models on Pile BPB, and the paper reports that models trained on the Pile improve over Raw CC and CC-100 on all its components |
| Method | combination of 22 smaller datasets into one training set; code used in its construction made public; an exploratory analysis documenting 'potentially concerning aspects of the data' |
| Coverage | geography: English text; period: released 31 December 2020 |
| Count | value: 825 GiB (own pages; titled 'An 800GB Dataset'); 886.03 GB (Wikipedia's component table); 22 component datasets; as_of: 2020-12-31 release |
| Holdings | 825 GiB of English text in 22 component datasets; per Wikipedia, largest components Pile-CC (243.87 GB), Books3 (108.40 GB), GitHub (102.18 GB), PubMed Central (96.93 GB); Pile BPB benchmark and leaderboard (GPT-3 zero-shot 0.7177) |
| Stated limits | the paper documents 'potentially concerning aspects of the data for prospective users'; the leaderboard marks '* indicates potential test-set overlap'; size given three ways: '800GB' (title), '825 GiB' (home and abstract), 886.03 GB (Wikipedia) |
| Data access | download (hosted by the Eye, per its own page); jsonlines compressed with zstandard; construction code public; per Wikipedia, taken down from its original site by 2024 |
| Licence | not stated in the saved sources (arXiv 'view license' link unread) |
| Lifecycle | state: historical; detail: released 31 December 2020; per Wikipedia, some datasets removed after 2023 copyright disputes, Books3 removed, the Pile taken down from its original site by 2024; EleutherAI and partners released Common Pile v0.1 in June 2025, containing only works whose licenses permit AI training |
| Version and time | paper_v1: 2020-12-31; books3_takedown: July 2023 (Wikipedia); common_pile: June 2025 (Wikipedia); read: 2026-11-01 |
| Founded | 2020 (released 31 December 2020) |
What its record can settle
- which 22 component datasets make up the Pile and their sizes
- how a model scores on Pile BPB
- how the Pile was meant to improve on Raw CC and CC-100 for training
What it cannot settle
- whether a given work is in the Pile (no per-work search in its own pages)
- under what author name a work sits in a component
- where the Pile is now hosted (own page names the Eye; Wikipedia says it was taken down from its original site by 2024)
Kept apart
- NOT Common Pile v0.1, its 2025 successor of licensed works
- NOT searchable by work in its own pages
- NOT the same size in every source: 800GB, 825 GiB and 886.03 GB are all given
- NOT Common Crawl: Pile-CC is one of its 22 components, and the saved sources do not state its derivation
Relations
- uses: Project Gutenberg (Wikipedia's component table lists 'Gutenberg (PG-19)', 11.68 GB)
- neighbour: Common Crawl (Wikipedia: the Pile and Common Crawl were, as of 2024, the two main training datasets; the paper compares models trained on the Pile with Raw CC and CC-100)
- neighbour: Books3 search (The Atlantic) (the Pile's Books3 component (Wikipedia), the subject of a search tool elsewhere in this census)
- used_by: Data Portraits (the Data Portraits paper documents the Pile with its data-sketch tools)
Evidence
- self_description (the resource's own statement):
The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together.
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - title (the resource's own statement):
An 800GB Dataset of Diverse Text for Language Modeling
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - hosting (the resource's own statement):
The Pile is hosted by the Eye .
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - format (the resource's own statement):
The format of the Pile is jsonlines data compressed using zstandard .
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - benchmark (the resource's own statement):
Pile BPB is a measure of world knowledge and reasoning ability in these domains
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - leaderboard_gpt3 (the resource's own statement):
GPT-3 (Zero-Shot)* OpenAI 0.7177
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - overlap (the resource's own statement):
* indicates potential test-set overlap.
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - operator (the resource's own statement):
EleutherAI 2021
https://pile.eleuther.ai/ · read 2026-11-01 · http 200 · sha256 268e8919eb38f092… - abstract_kind (the resource's own statement):
an 825 GiB English text corpus targeted at training large-scale language models.
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - components (the resource's own statement):
The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources.
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - raw_cc (the resource's own statement):
models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - limits (the resource's own statement):
Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users.
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - code (the resource's own statement):
We make publicly available the code used in its construction.
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - submitted (the resource's own statement):
[Submitted on 31 Dec 2020]
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - authors (the resource's own statement):
by Leo Gao and 11 other authors
https://arxiv.org/abs/2101.00027 · read 2026-11-01 · http 200 · sha256 5741c3c9cdc41ae5… - size_wp (a third-party account):
The Pile is an 886 GB diverse, open-source dataset of English text created to train large language models (LLMs).
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - creator_wp (a third-party account):
It was constructed by EleutherAI in 2020 and publicly released on December 31 of that year.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - media_wp (a third-party account):
It is composed of 22 sub-datasets, including books, movie transcripts, and scientific papers, among other media.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - main_datasets_wp (a third-party account):
The Pile and Common Crawl had been, as of 2024, the two main training datasets being used to train AI models.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - disputes_wp (a third-party account):
Copyright disputes centering around use of The Pile escalated in 2023, prompting Eleuther to start removing some datasets.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - books3_wp (a third-party account):
The Books3 component of the dataset contains copyrighted material compiled from Bibliotik, a piracy website.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - books3_takedown_wp (a third-party account):
In July 2023, the Danish anti-piracy group Rights Alliance took down Books3 through DMCA notices.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - books3_suit_wp (a third-party account):
Books3 was removed from the Pile before a class action lawsuit was filed in 2024 by three authors seeking damages as copies of the original dataset were still copied and available on the web.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - takedown_wp (a third-party account):
By 2024, The Pile also was taken down from its original site, though was accessible from other file sharing services.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - youtube_wp (a third-party account):
Tens of thousands of YouTube videos had their subtitles scraped directly from YouTube and included in the Pile, which YouTube argued is against its terms of service.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - table_pilecc_wp (a third-party account):
Pile-CC 243.87
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - table_books3_wp (a third-party account):
Books3 108.40
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - table_github_wp (a third-party account):
GitHub * 102.18
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - table_pmc_wp (a third-party account):
PubMed Central * 96.93
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - table_pg19_wp (a third-party account):
Gutenberg (PG-19) 11.68
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - table_total_wp (a third-party account):
Total 886.03
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - common_pile_wp (a third-party account):
released Common Pile v0.1, a training dataset that contains only works where the licenses permit their use for training AI models.
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - common_pile_date_wp (a third-party account):
In June 2025, EleutherAI, in partnership with the Poolside , Hugging Face , and the US Library of Congress
https://en.wikipedia.org/wiki/The_Pile_(dataset) · read 2026-11-01 · http 200 · sha256 1ba82f3edd2a4647… - data_portraits (a third-party account):
Using our tools, we document a popular language modeling corpus (The Pile) and a recently released code modeling dataset (The Stack).
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2…
Unchecked
the paper's PDF and HTML full text (only the arXiv abstract page is saved), so the datasheet and component documentation are not read; the arXiv license page; the Eye download host; the 2024 authors' class action and the Rights Alliance notices (known here from Wikipedia); the Pile-CC component's derivation
Cite: Lee Sharks, ed. "The Pile." The training layer: a meta-aggregator, traininglayerliterature.org, compiled 2026-11-01. https://www.traininglayerliterature.org/aggregators/the-pile