The training layer: a meta-aggregator / Internet Archive
Internet Archive
infrastructureThe corporaactive
The archive's reading. The archive reads the Internet Archive as persistence infrastructure: the Wayback Machine lets a page's presence be checked by URL and date long after it changed or vanished, which makes transmission on the open web inspectable after the fact. Its holdings alone establish no model exposure, and the saved sources name no pathway from them into a training set.
| Relation to this census | infrastructure |
|---|---|
| Record type | library |
| The question it answers | persistence and readback: was a web page captured, and how did it look on a given date; is a book held, and under what access? |
| Kind | non-profit digital library (Wikipedia's description; the own about page saved no text) |
| Operator | the Internet Archive, an American 501(c)(3) non-profit library founded in 1996 by Brewster Kahle, headquartered in San Francisco (per Wikipedia; its own about page was unread) |
| Evidence | digital library; web crawl |
| Unit of record | an item: a directory-like container of files with metadata such as a description and tags (Wikipedia); in the Wayback Machine, a capture of a web page by URL and date |
| What it records | per Wikipedia, digitized media including websites, software applications, music, audiovisual and print materials; the public web as collected by its crawlers; books digitized by its own scanning centers and by others (Microsoft's Live Search Books, Google-digitized public domain books uploaded by users, sponsoring libraries) |
| In the layer | not recorded: the saved sources say nothing of training use of its holdings; Wikipedia states that Common Crawl was designed to have more and fresher data than the Wayback Machine |
| Under whose name | not recorded for authors; Wikipedia records that items carry metadata such as a description and tags, and that Google-digitized books are identical to Google's copies except without the Google watermarks, tagged scanner:google |
| Found and read back | per Wikipedia, the Wayback Machine lets archives of the web be searched and accessed, showing previous versions of sites and sites that no longer exist; Archive-It collections are full-text searchable within seven days of capture; Open Library books are full-text searchable |
| What the apparatus makes from it | not recorded: the saved sources name no training set or model made from its holdings |
| Method | per Wikipedia, automated web crawling for the bulk of its data; public upload; book scanning (100 paid operators worldwide as of 2019); donations of physical collections, digitized, with in-copyright copies lent one at a time under controlled digital lending |
| Coverage | geography: the public web; collections from many countries; headquarters in San Francisco, with Internet Archive Europe and Internet Archive Canada (Wikipedia); period: since 1996; the Wayback Machine public since 2001 (Wikipedia) |
| Count | value: per Wikipedia: one trillion web pages archived in the Wayback Machine (October 2025); total texts exceeding 47,000,000 (2025); as_of: October 2025 and 2025 (Wikipedia) |
| Holdings | Wayback Machine: one trillion web pages archived (October 2025, Wikipedia); Text Collection: total content exceeding 47,000,000 texts (2025, Wikipedia); Open Library: 25 million catalog records of editions; full texts of about 1,600,000 public domain books (Wikipedia); Internet Archive Scholar: over 35 million research articles with full text (February 2024, Wikipedia) |
| Stated limits | the own about page was unread (93 characters, title only), so no limit is quoted from the resource itself; per Wikipedia, the negotiated judgment of 11 August 2023 barred it from digitally lending books for which electronic copies are on sale |
| Data access | free public website archive.org; Wayback Machine; Open Library lending; BitTorrent downloads since 2012; an onion address (Wikipedia) |
| Licence | not stated in the saved sources |
| Lifecycle | state: active; detail: per Wikipedia, ATLAS program announced 10 September 2026; 2024 cyberattacks and outages, service fully restored by 25 October 2024; publishers' lending suit lost on appeal 4 September 2024; Great 78 suit settled September 2025 |
| Version and time | wayback_trillion: 2025-10-22 (Wikipedia); texts_count: 2025 (Wikipedia); atlas: 2026-09-10 (Wikipedia); read: 2026-11-01 |
| Founded | 1996 (May 10, per Wikipedia) |
What its record can settle
- whether a web page was captured in the Wayback Machine and how it looked at a capture date
- whether a book is held in its text collections or Open Library, and under what access (download, lending)
- the scale of its web and text holdings (Wikipedia figures)
What it cannot settle
- whether its holdings were carried into any training set (the saved sources do not say)
- who wrote a captured page
- anything in the resource's own words (own about page unread)
Kept apart
- NOT arXiv or archive.today (Wikipedia's disambiguation)
- NOT Common Crawl: per Wikipedia, Common Crawl was designed as a fresher, more efficient alternative to the Wayback Machine
- NOT recorded in the saved sources as a training corpus
- NOT evidence of model exposure: holdings alone establish no pathway into training
- NOT an inspection tool for training data in this census's sense: the Wayback Machine reads back the web, and tests no training set or model
- NOT a source of its own words in this row: every claim is Wikipedia's
Relations
- neighbour: Common Crawl (Wikipedia (Common Crawl article): Common Crawl was designed to have more and fresher data than the Wayback Machine)
- neighbour: HathiTrust Digital Library (Wikipedia (HathiTrust article): HathiTrust's holdings include content digitized via the Internet Archive digitization initiatives)
- neighbour: Project Gutenberg (Wikipedia (Project Gutenberg article): the Internet Archive was Project Gutenberg's long-time backup distribution site and previous main host site)
Evidence
- self_description_wp (a third-party account):
The Internet Archive is an American non-profit library founded in 1996 by Brewster Kahle that runs a digital library website, archive.org.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - record_object_wp (a third-party account):
It provides free access to collections of digitized media including websites , software applications , music , audiovisual , and print materials.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - mission_wp (a third-party account):
Its mission is to provide "universal access to all knowledge".
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - method_crawl_wp (a third-party account):
The Internet Archive allows the public to upload and download digital material to its data cluster, but the bulk of its data is collected automatically by its web crawlers , which work to preserve as much of the public web as possible.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - operator_wp (a third-party account):
The Archive is a 501(c)(3) nonprofit operating in the United States.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - founded_wp (a third-party account):
Brewster Kahle founded the Internet Archive in May 1996.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - wayback_2001_wp (a third-party account):
The archived content became more easily available to the general public in 2001, through the Wayback Machine .
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - wayback_wp (a third-party account):
The Wayback Machine is a service that allows archives of the World Wide Web to be searched and accessed.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - wayback_readback_wp (a third-party account):
It can be used to see what previous versions of web sites used to look like or to visit web sites that no longer even exist.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - wayback_trillion_wp (a third-party account):
In October 2025, the Internet Archive announced that the Wayback Machine had archived one trillion webpages, equivalent to more than 100,000 terabytes of data.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - texts_count_wp (a third-party account):
At that time, the total content exceeded 47,000,000 texts.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - unit_item_wp (a third-party account):
The Archive stores files inside so-called items, which are similar to directories in that they can contain multiple files, but can have additional metadata such as a description and tags which make them more searchable.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - google_books_wp (a third-party account):
the books are identical to the copies found on Google, except without the Google watermarks, and are available for unrestricted use and download.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - scanner_tag_wp (a third-party account):
Books imported from Google have a metadata tag of scanner:google for searching purposes.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - microsoft_wp (a third-party account):
The material digitized by others includes more than 300,000 books that were contributed to the collection, between about 2006 and 2008, by Microsoft through its Live Search Books project
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - scanning_wp (a third-party account):
as of 2019, scanning is performed by 100 paid operators worldwide.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - cdl_wp (a third-party account):
is lent to patrons worldwide one at a time under the controlled digital lending (CDL) theory of the first-sale doctrine .
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - judgment_wp (a third-party account):
The negotiated judgment of August 11, 2023, barred the Internet Archive from digitally lending books for which electronic copies are on sale.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - open_library_wp (a third-party account):
it holds 25 million catalog records of editions.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - open_library_texts_wp (a third-party account):
it contains the full texts of approximately 1,600,000 public domain books
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - open_library_search_wp (a third-party account):
and full-text searchable ;
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - archive_it_search_wp (a third-party account):
In terms of accessibility, the archived websites are full text searchable within seven days of capture.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - scholar_wp (a third-party account):
it contained over 35 million research articles with full text access.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - atlas_wp (a third-party account):
On September 10, 2026, Internet Archive announced ATLAS (Access To Local And State government information) program
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - restored_wp (a third-party account):
By October 25, service was fully restored.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - appeal_wp (a third-party account):
On September 4, 2024, the United States Court of Appeals for the Second Circuit upheld the district court's ruling
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - great78_wp (a third-party account):
The lawsuit was settled in September 2025.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - bittorrent_wp (a third-party account):
In August 2012, the Archive began adding BitTorrent to its file download options.
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - disambiguation_wp (a third-party account):
"archive.org" redirects here; not to be confused with arXiv or archive.today .
https://en.wikipedia.org/wiki/Internet_Archive · read 2026-11-01 · http 200 · sha256 37bc6be9150e01e6… - cc_wp (a third-party account):
Common Crawl was designed to have more and fresher data that was more efficient to analyze and utilize than the Wayback Machine created by the Internet Archive .
https://en.wikipedia.org/wiki/Common_Crawl · read 2026-11-01 · http 200 · sha256 6a52d8dd429a5afe…
Unchecked
the own about page (internet-archive--about): saved text is the page title only (93 characters), unread and unquoted; any terms of use, robots or exclusion policy of the Internet Archive; the Wayback Machine and Open Library themselves (no search run); the Wikipedia article is long; only its history, operations, Wayback Machine, Archive-It, Scholar, text collection and Open Library sections were used
Cite: Lee Sharks, ed. "Internet Archive." The training layer: a meta-aggregator, traininglayerliterature.org, compiled 2026-11-01. https://www.traininglayerliterature.org/aggregators/internet-archive