The training layer: a meta-aggregator / Common Crawl
Common Crawl
infrastructureThe corporaInspectionactive
The archive's reading. The archive reads Common Crawl as the receiving apparatus's first intake, the plainest case of #186's persistence criterion: the place 'where the training and retrieval layers actually look', entered by being on the open web and addressed by URL. What is made from a page (refined datasets, trained models) is made downstream by others, as Wikipedia reports; crawl inclusion can be inspected in the URL Index, use of a captured page in a given model's training is a further claim its own pages do not make, and Wikipedia's report of The Atlantic's finding marks the limit of the index as a witness. A work's name survives there only as far as it is written into the page itself.
| Relation to this census | infrastructure |
|---|---|
| Record type | dataset |
| The question it answers | training membership at the source: was a given URL captured in a given crawl, the raw data from which others build training datasets? |
| Kind | free, open repository of web crawl data |
| Operator | the Common Crawl Foundation, a 501(c)(3) non-profit founded in 2007; Wikipedia names Gil Elbaz as founder and Rich Skrenta as executive director |
| Evidence | web crawl; dataset |
| Unit of record | a crawl (named by release, e.g. CC-MAIN-2026-43) holding captured web pages as raw page data, metadata extracts and text extracts; a page is addressed by its URL in the URL Index |
| What it records | pages of the open web, crawled each month: over 300 billion pages spanning 15 years, with 3 to 5 billion new pages added each month |
| In the layer | whatever its crawler captured from the open web; Wikipedia reports that AI companies began training large language models on the data in the 2020s, that ChatGPT as of 2023 was largely trained on it, and that refined datasets (FineWeb, DCLM, C4) are built from its raw data |
| Under whose name | not recorded: the saved sources state no author or attribution field; a page is held by URL, with its metadata extracts and text extracts |
| Found and read back | a page can be searched by URL in the Common Crawl URL Index (also a CDXJ Index); the corpus can be downloaded whole or in part. Wikipedia reports The Atlantic's November 2025 finding that the public search function showed no entries for sites that had requested removal while those sites were still in the scrapes |
| What the apparatus makes from it | not recorded on its own pages; per Wikipedia, other organizations refine its raw data into datasets (FineWeb, DCLM, C4), and AI companies train large language models on the data |
| Method | automated web crawling by its crawler, CCBot; crawls published as numbered releases; per Wikipedia, the crawl began from a list of URLs donated by the search engine Blekko |
| Coverage | geography: the open web (no geography stated); period: crawling since 2007 (home page) or regularly collected since 2008 (overview); crawls published since 2011 (Wikipedia); latest listed crawl CC-MAIN-2026-43 |
| Count | value: over 300 billion pages spanning 15 years; petabytes of data; 3-5 billion new pages each month; as_of: home page as read 2026-11-01 |
| Holdings | over 300 billion pages spanning 15 years; 3-5 billion new pages added each month; raw web page data, metadata extracts and text extracts; host- and domain-level web graphs (latest: August-October 2026); cited in over 10,000 research papers |
| Stated limits | the saved own pages state no limit on what is crawled; 'Access to the corpus hosted by Amazon is free' (Wikipedia adds that users may incur storage and compute costs); its own pages say 'since 2007' (home) and 'regularly collected since 2008' (overview) |
| Data access | free on Amazon Web Services' Public Data Sets and on academic cloud platforms; analysis jobs on Amazon or download whole or in part; URL Index and CDXJ Index; per Wikipedia, experimental distribution through Hugging Face Storage Bucket from April 2026 |
| Licence | not stated in the saved own pages (Terms of Use unread); Wikipedia's infobox gives 'Apache 2.0 (software)' with a clarification-needed flag |
| Lifecycle | state: active; detail: monthly crawls; latest listed CC-MAIN-2026-43; October 2026 web graph announced on the home page |
| Version and time | latest_crawl: CC-MAIN-2026-43; web_graph: August-October 2026; atlantic_investigation: November 2025 (Wikipedia); read: 2026-11-01 |
| Founded | 2007 |
What its record can settle
- whether a given URL was captured in a given crawl (URL Index)
- the size and cadence of the crawl
- which crawl releases exist
What it cannot settle
- who wrote a captured page, or under what name
- which models were trained on which crawl (the training claims in this row are Wikipedia's)
- whether a site that asked for removal is out of the data (Wikipedia reports The Atlantic's finding that it was not)
Kept apart
- NOT a library of authored works: its unit is a crawled page held by URL
- NOT a training dataset itself in its own words: it is a repository of crawl data, and refined datasets (C4, FineWeb, DCLM) are built from it by others
- NOT the Internet Archive's Wayback Machine (Wikipedia: designed to have more and fresher data than it)
- NOT shown by its saved sources to remove a page on request (Wikipedia reports The Atlantic's contrary finding)
- NOT evidence that a captured page was used in a given model's training: crawl inclusion and demonstrated training use are separate claims
Relations
- neighbour: Internet Archive (Wikipedia: Common Crawl was designed to have more and fresher data than the Internet Archive's Wayback Machine)
- neighbour: The Pile (Wikipedia: by 2024 The Pile and Common Crawl were the two main training datasets; the saved sources do not state what the Pile's Pile-CC component is drawn from)
Evidence
- self_description (the resource's own statement):
Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - operator (the resource's own statement):
Common Crawl is a 501(c)(3) non–profit founded in 2007.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - purpose (the resource's own statement):
We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - count (the resource's own statement):
Over 300 billion pages spanning 15 years.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - since_2007 (the resource's own statement):
Free and open corpus since 2007.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - citations (the resource's own statement):
Cited in over 10,000 research papers.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - cadence (the resource's own statement):
3–5 billion new pages added each month.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - web_graph (the resource's own statement):
We are extremely pleased to announce the release of our October 2026 Web Graph, comprising data from August, September, and October 2026.
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - petabytes (the resource's own statement):
The Common Crawl corpus contains petabytes of data, regularly collected since 2008.
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - latest_crawl (the resource's own statement):
CC-MAIN-2026-43
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - unit (the resource's own statement):
The corpus contains raw web page data, metadata extracts, and text extracts.
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - storage (the resource's own statement):
Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world.
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - access (the resource's own statement):
Access to the corpus hosted by Amazon is free.
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - download (the resource's own statement):
You may use Amazon’s cloud platform to run analysis jobs directly against it or you can download it, whole or in part.
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - readback (the resource's own statement):
You can search for pages in our corpus using the Common Crawl URL Index.
https://commoncrawl.org/overview · read 2026-11-01 · http 200 · sha256 40dc1c9a6e578a2c… - indexes (the resource's own statement):
Overview CDXJ Index URL Index Web Graphs Latest Crawl Crawl Stats Graph Stats Errata
https://commoncrawl.org/ · read 2026-11-01 · http 200 · sha256 34837e49010690a9… - founder (a third-party account):
Common Crawl was founded by Gil Elbaz .
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - director (a third-party account):
Rich Skrenta is the executive director.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - license_wp (a third-party account):
Apache 2.0 (software) [ clarification needed ]
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - costs_wp (a third-party account):
Access to the data is free on Amazon Web Services , but users may incur storage and compute costs.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - publishing_2011 (a third-party account):
It began publishing its crawls in 2011.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - blekko (a third-party account):
which started by crawling a list of URLs donated by the search engine Blekko .
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - in_layer_llm (a third-party account):
The data had mostly been primarily used by researchers and some startups until the 2020s, when AI companies started training large language models using the data.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - in_layer_chatgpt (a third-party account):
ChatGPT , as of 2023, was largely trained on Common Crawl,
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - pile_wp (a third-party account):
By 2024, The Pile and Common Crawl had been the two main training datasets being used to train AI models.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - atlantic (a third-party account):
It included misleading results in the public search function on its website that showed no entries for websites that had requested their archives be removed, when in fact those sites were still included in its scrapes used by AI companies.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - atlantic_date (a third-party account):
In November 2025, an investigation by technology journalist Alex Reisner for The Atlantic revealed that Common Crawl misled publishers when it claimed it respected paywalls in its scraping
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - ai_funding (a third-party account):
In 2023, it began receiving significant financial support from AI companies, including Anthropic and OpenAI , each of which donated $250,000.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - ccbot_blocked (a third-party account):
As of 2025, Reisner found that CCBot was the most widely-blocked bot by the top 1000 websites.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - refined (a third-party account):
A number of organizations take raw Common Crawl data and refine it into datasets that exclude edgy content or are otherwise higher-quality for their purposes, such as FineWeb, DCLM and C4.
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - wayback_wp (a third-party account):
Common Crawl was designed to have more and fresher data that was more efficient to analyze and utilize than the Wayback Machine created by the Internet Archive .
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe… - hugging_face_wp (a third-party account):
In April 2026, Common Crawl experimentally began to distribute its data through Hugging Face Storage Bucket , in addition to its standard storage on Amazon S3 .
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe…
Unchecked
the CCBot, Opt-Out Ledger, Errata, FAQ, Terms of Use and About pages (named in the navigation only); the URL Index and CDXJ Index themselves (no query run); The Atlantic's investigation itself (known here from Wikipedia); Wikipedia's claims on which models were trained on the data
Cite: Lee Sharks, ed. "Common Crawl." The training layer: a meta-aggregator, traininglayerliterature.org, compiled 2026-11-01. https://www.traininglayerliterature.org/aggregators/common-crawl