The training layer: a meta-aggregator / Data Portraits
Data Portraits
https://arxiv.org/abs/2303.03919
infrastructureInspectionThe corporahistorical
The archive's reading. A readback instrument held on the corpus side: a portrait records the dataset so that a work can be tested for membership, which settles #186's 'is the work in the layer' for a recorded corpus. Dataset membership is not proof that a particular model used a passage, and the name and what a model made from the work (#1543's question) are outside what the saved abstract records.
| Relation to this census | infrastructure |
|---|---|
| Record type | method |
| The question it answers | Is a given example in a recorded training dataset (training membership)? |
| Kind | artifacts that record training data and allow downstream inspection, proposed for widespread adoption and implemented with data sketching, in a research paper (NeurIPS 2023 Datasets and Benchmarks) |
| Operator | the paper's two authors, Marc Marone and Benjamin Van Durme; the saved source names no institution, and the live interface's operator is unread |
| Evidence | research paper; dataset documentation |
| Unit of record | a given example, queried against a portrait (a data sketch) of a training dataset |
| What it records | the contents of training datasets, recorded so that a query can answer whether a model has already encountered a given example during training; implemented for The Pile and The Stack |
| In the layer | a portrait records a training dataset; the paper documents a popular language modeling corpus (The Pile) and a code modeling dataset (The Stack) |
| Under whose name | not recorded; the saved abstract does not say whether a portrait keeps any author, title or source metadata for a recorded example |
| Found and read back | a membership query against the recorded dataset: the paper poses 'has the model already encountered a given example during training?' and stresses 'fast and space efficient querying'; a live interface is released (unread) |
| What the apparatus makes from it | not recorded as a product of the model; the paper states the tool 'enables answering questions about test set leakage and model plagiarism', which checks material against the recorded training data |
| Method | data sketching: a compact record of a dataset built for fast, space-efficient membership queries, costing about 3% of the dataset size; the paper also outlines the properties such an artifact should have and how existing solutions can increase transparency |
| Coverage | geography: not recorded; period: not recorded; the datasets recorded are The Pile and The Stack |
| Count | value: two datasets documented (The Pile and The Stack); overhead about 3% of the dataset size; as_of: arXiv v2, 14 December 2023 |
| Holdings | portraits of The Pile and The Stack; a sketch-based implementation costing 3% of the dataset size in overhead; a released live interface (unread); a call on dataset and model creators to release Data Portraits |
| Stated limits | the saved abstract states no limits of the method; its framing is that training datasets are 'increasingly immense and opaque' |
| Data access | the paper on arXiv (PDF, HTML, TeX source); the live interface linked as 'this https URL' is unread |
| Licence | not recorded in the saved source (the arXiv page links a 'view license' for the paper) |
| Lifecycle | state: historical; detail: a published paper: v1 6 March 2023, v2 14 December 2023 (NeurIPS 2023 Datasets and Benchmarks); whether the live interface is still served is unread |
| Version and time | paper_v1: 2023-03-06; paper_v2: 2023-12-14; venue: NeurIPS 2023 Datasets and Benchmarks; read: 2026-11-01 |
| Founded | 2023 (arXiv v1, 6 March 2023) |
What its record can settle
- whether a given example is in a recorded dataset (The Pile, The Stack)
- questions of test set leakage and model plagiarism against a recorded dataset, as the paper states
What it cannot settle
- whether a particular model's output depends causally on a recorded passage
- under whose name a recorded example sits in the dataset
- anything about datasets that have no portrait
Kept apart
- NOT a corpus: it records corpora built by others
- NOT proof of a model's causal use of a passage: it records membership of a dataset
- NOT a model-side test: it queries the recorded data and needs a portrait of it
- NOT a record of authorship: the saved abstract says nothing of attribution metadata
- NOT the live interface: the paper is the object read; the interface is a linked object, unread
Relations
- uses: The Pile (the paper documents The Pile with its tools ('a popular language modeling corpus (The Pile)'))
- neighbour: OLMoTrace (both check text against training data from outside the model's weights: Data Portraits by membership queries on a recorded corpus, OLMoTrace by verbatim span matching against the full corpus)
Evidence
- title (the resource's own statement):
Data Portraits: Recording Foundation Model Training Data
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - authors (the resource's own statement):
Authors: Marc Marone , Benjamin Van Durme
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - opacity (the resource's own statement):
Foundation models are trained on increasingly immense and opaque datasets.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - question (the resource's own statement):
it can be difficult to answer the straightforward question: has the model already encountered a given example during training?
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - self_description (the resource's own statement):
We therefore propose a widespread adoption of Data Portraits: artifacts that record training data and allow for downstream inspection.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - properties (the resource's own statement):
First we outline the properties of such an artifact and discuss how existing solutions can be used to increase transparency.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - method (the resource's own statement):
We then propose and implement a solution based on data sketching, stressing fast and space efficient querying.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - datasets (the resource's own statement):
Using our tools, we document a popular language modeling corpus (The Pile) and a recently released code modeling dataset (The Stack).
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - answers (the resource's own statement):
We show that our solution enables answering questions about test set leakage and model plagiarism.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - overhead (the resource's own statement):
Our tool is lightweight and fast, costing only 3% of the dataset size in overhead.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - interface (the resource's own statement):
We release a live interface of our tools at this https URL and call on dataset and model creators to release Data Portraits as a complement to current documentation practices.
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - venue (the resource's own statement):
NeurIPS 2023 Datasets and Benchmarks
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - v1 (the resource's own statement):
[v1] Mon, 6 Mar 2023 04:22:33 UTC (651 KB)
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2… - v2 (the resource's own statement):
[v2] Thu, 14 Dec 2023 16:55:42 UTC (614 KB)
https://arxiv.org/abs/2303.03919 · read 2026-11-01 · http 200 · sha256 0a499acc8164a2d2…
Unchecked
the live interface (linked as 'this https URL' and unsaved): what it lets a user query, and whether it is still served; the paper's full text beyond the abstract: the sketch's error properties, what a query returns, and any attribution or metadata fields
Cite: Lee Sharks, ed. "Data Portraits." The training layer: a meta-aggregator, traininglayerliterature.org, compiled 2026-11-01. https://www.traininglayerliterature.org/aggregators/data-portraits