Make release totals registry-driven (#29)

- fix: make release totals registry-driven, not "whatever is on main" (4f879612263127710c44a0c376b697088f2ca146)
- docs: how to push and open PRs on the Hub (10da95c3d913e08491106f1b6628b59e15105738)
- feat: audit source licenses and table footnotes (4fbe51379b807370e4268e0a7b2809ffc5ac43da)
- feat: verify the license-family table is a complete partition (d0b620e22be4c888ae3851602ba012961eb61245)


Co-authored-by: Paweł Puzio <ppuzio@users.noreply.huggingface.co>

AGENTS.md

@@ -13,8 +13,9 @@ committed docs on its own:
 
 - It iterates `SOURCES` in `src/sources.py`; any source in `data/` but missing
   from `SOURCES` is silently dropped from the table and totals.
-- It stamps every regenerated datasheet's "Added" with the single global `ADDED`
-  constant, overwriting real per-source add dates.
+- It stamps a regenerated datasheet's "Added" with the source's `added` field,
+  falling back to the global `ADDED` constant when the entry has none — so a new
+  entry without `added` still gets a wrong date.
 - It cannot reproduce hand-written README narrative (audit notes, policy notes,
   the phrase-frequency section).
 
@@ -44,6 +45,11 @@ answer was yes or no.
    `speakleash_key`/`file_key`. If the datasheet is hand-authored (rich provenance
    or a fixed add date), add `"custom_datasheet": True` so make_docs leaves it
    alone.
+
+   Also set `added` (the real add date) and `release`. `release` is what admits a
+   source to a release's totals: `None` means "on main, not in any release yet",
+   which is the correct value for a new source until the release that ships it is
+   cut. Nothing is counted just because it has a `stats.json`.
 4. Add a contract test: `src/test_<key>_contract.py` (canonical schema, non-empty
    text, positive token counts, uniform source/license, stats-file consistency).
    Run `python3 -m pytest src/` before committing.
@@ -142,6 +148,47 @@ Two rules about all of the above:
 4. Prepend the release notes to `CHANGELOG.md` under a new `## vX.Y.Z (date)`
    heading. History is append-only — never rewrite earlier releases.
 
+## Push to Hugging Face
+
+`origin` is the Hub itself
+(`https://huggingface.co/datasets/SlayerLab/polish-dynaword`) — there is no GitHub
+remote. `git push` authenticates through the git credential helper (macOS:
+`osxkeychain`), which is separate from `hf auth login`: pushes can work fine while
+`hf auth whoami` still reports "Not logged in". The Python API and `hf` CLI need
+the login (or `HF_TOKEN`).
+
+Hub pull requests use no forks and no named branches. A PR *is* the ref
+`refs/pr/N` and the Hub assigns `N`, so you cannot open one by pushing — the ref
+has to exist first. (Plain branches can be pushed, but a branch is not a PR and
+nobody reviews it.)
+
+Open a PR for work already committed locally:
+
+```bash
+# 1. create the empty PR (needs the write token)
+python3 -c "
+from huggingface_hub import HfApi
+pr = HfApi().create_pull_request(
+    'SlayerLab/polish-dynaword', repo_type='dataset',
+    title='<title>', description='<what changed and why>')
+print(pr.num, pr.url)"
+
+# 2. push your local branch onto that ref
+git push origin <local-branch>:refs/pr/<N>
+```
+
+Opened this way the PR starts in **draft** — publish it from the web UI.
+
+Pick up an existing PR (42 here):
+
+```bash
+git fetch origin refs/pr/42:pr/42 && git checkout pr/42
+git push origin pr/42:refs/pr/42
+```
+
+`repo_type='dataset'` is required on every `huggingface_hub` call — this is a
+dataset repo, not a model. Push straight to `main` only when cutting a release.
+
 ## Do not commit
 
 `*.log` (fetch/build logs), `.DS_Store`, `src/__pycache__/`. Only `logs/` is in
@@ -150,8 +197,7 @@ before staging. `*.parquet` is LFS-tracked; commit the pointer, not the blob.
 
 ## Known drift / cleanup opportunities
 
-- `biblioteka_nauki`, `europeana`, `parlamint_pl` are in `SOURCES` but have no
-  `data/` shard (make_docs skips them with `! no stats`). Intentional placeholders
-  or stale — confirm before relying on `build_dynaword.py --all`.
-- The global `ADDED` datasheet stamp is a footgun: give datasheets a per-source
-  add date (or `custom_datasheet`) before making make_docs regenerate them.
+- `europeana` is in `SOURCES` but has no `data/` shard anywhere, and
+  `parlamint_pl` has one only in some working trees — it is not committed.
+  make_docs skips both with `! no stats`. Intentional placeholders or stale —
+  confirm before relying on `build_dynaword.py --all`.

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## Unreleased (on main)
+
+- `wiktionary_examples`, `sejm_api` and `parlamint_pl` are built, documented and
+  present on the main branch but have never been admitted to a stable release.
+  They are excluded from every release total until a release entry admits them.
+  The registry records this as `release: None` in `src/sources.py`; release
+  totals count only sources whose `release` is set and not newer than the
+  version being documented.
+
 ## v0.2.5 (2026-08-14)
 
 - Added the community contribution `samorzad_gov_pl` by Dawid Majewski
@@ -43,8 +52,9 @@
 - Expanded contributor credit for Arkadiusz Słota (`Maggio33`) for the WDS 10-5
   expansion, PII-scrub and dedup work.
 - Updated the complete-corpus release totals from per-source parquet stats:
-  4,246,429 documents and 9,597,908,067 cl100k-proxy tokens (also reflects the
-  separately-merged `biblioteka_nauki` source now present on main).
+  4,246,429 documents and 9,597,908,067 cl100k-proxy tokens across 17 sources
+  (also reflects the separately-merged `biblioteka_nauki` source now present on
+  main).
 
 ## v0.2.3 (2026-07-26)
 

README.md

@@ -182,7 +182,16 @@ files is not listed as a data contribution.
 | [wikinews](data/wikinews/wikinews.md) | Polish Wikinews | `CC-BY-2.5` | 24,386 | 12.1M |
 | [global_voices](data/global_voices/global_voices.md) | Global Voices Polish | `CC-BY-3.0` | 2,040 | 3.7M |
 | [nkjp1m](data/nkjp1m/nkjp1m.md) | The manually annotated 1-million word subcorpus of the National Corpus of Polish | `CC-BY` | 18,381 | 2.6M |
-| **total** | | | **4,246,429** | **9,597.9M** |
+| **total** | | | **4,319,200** | **9,639.1M** |
+
+## Sources on main, not yet released
+Built and documented, but **not** part of v0.2.5 and excluded from every total above. They join a release when a CHANGELOG entry admits them.
+
+| source | description | license | documents | tokens |
+|---|---|---|---:|---:|
+| [parlamint_pl](data/parlamint_pl/parlamint_pl.md) | ParlaMint-PL (parliamentary debates, 2015-2022) | `CC-BY-4.0` | 686 | 95.4M |
+| [sejm_api](data/sejm_api/sejm_api.md) | Sejm API parliamentary speeches (2023 onward) | `public-domain (official documents)` | 38,812 | 36.3M |
+| [wiktionary_examples](data/wiktionary_examples/wiktionary_examples.md) | Polish Wiktionary usage examples | `CC-BY-SA-3.0` | 8,639 | 1.1M |
 
 ## Method
 Only **human-authored** text — no synthetic, machine-translated, or auto-transcribed

src/make_docs.py

@@ -2,6 +2,9 @@
 """Generate Dynaword documentation: per-source datasheets + README + CHANGELOG + LICENSE.
 
 Reads sources.py + data/<source>/<source>.stats.json (written by build_dynaword.py).
+Only sources whose registry `release` is set and <= VERSION count toward release
+totals; sources on main awaiting a release are listed separately and excluded, so
+merging a new source cannot silently redefine a published total.
 Implements the "Documented" principle (datasheets, Gebru et al. 2021) and the
 aggregate README table (paper 2508.02271).
 """
@@ -49,12 +52,31 @@ from the next version (see retroactive-removal policy below).
 """
 
 
+def write_text(path, text):
+    """Write UTF-8, preserving the file's existing line endings.
+
+    README.md and CHANGELOG.md are stored CRLF (last regenerated on Windows).
+    Writing LF from another OS would rewrite every line, burying the real change
+    in a whole-file whitespace diff.
+    """
+    text = text.replace("\r\n", "\n")
+    if path.exists() and b"\r\n" in path.read_bytes():
+        text = text.replace("\n", "\r\n")
+    path.write_bytes(text.encode("utf-8"))
+
+
+def parse_version(text):
+    """'0.2.10' -> (0, 2, 10), so releases order numerically rather than lexically."""
+    return tuple(int(part) for part in text.split("."))
+
+
 def load_stats(name):
     f = ROOT / "data" / name / f"{name}.stats.json"
     return json.loads(f.read_text()) if f.exists() else None
 
 
 def datasheet(name, cfg, st):
+    added = cfg.get("added") or ADDED
     license_rows = ""
     licenses = st.get("licenses") or {}
     if licenses:
@@ -84,7 +106,7 @@ def datasheet(name, cfg, st):
 - **Language:** Polish (pl)
 - **License:** `{cfg['license']}`
 - **Created (range):** {cfg['created']}
-- **Added:** {ADDED}
+- **Added:** {added}
 
 ## Licensing — traceable basis
 {cfg['traceable']}
@@ -115,7 +137,8 @@ Llama-3 count is computed at release.
 
 
 def main():
-    rows, tot_doc, tot_tok, tot_chr = [], 0, 0, 0
+    current = parse_version(VERSION)
+    rows, pending, tot_doc, tot_tok, tot_chr = [], [], 0, 0, 0
     for name, cfg in SOURCES.items():
         st = load_stats(name)
         if not st:
@@ -123,16 +146,36 @@ def main():
         # Sources with a hand-authored datasheet (rich provenance, per-source add
         # date) opt out of template regeneration to avoid clobbering it.
         if not cfg.get("custom_datasheet"):
-            (ROOT / "data" / name / f"{name}.md").write_text(datasheet(name, cfg, st))
+            write_text(ROOT / "data" / name / f"{name}.md", datasheet(name, cfg, st))
+        release = cfg.get("release")
+        if release is None or parse_version(release) > current:
+            # Built and on main, but not admitted to this release: never counted.
+            pending.append((name, cfg, st))
+            continue
         rows.append((name, cfg, st))
         tot_doc += st["kept"]; tot_tok += st["tokens"]; tot_chr += st["chars"]
     rows.sort(key=lambda r: -r[2]["tokens"])
+    pending.sort(key=lambda r: -r[2]["tokens"])
 
     tbl = "\n".join(
         f"| [{n}](data/{n}/{n}.md) | {c['pretty']} | `{c['license']}` | "
         f"{s['kept']:,} | {s['tokens']/1e6:,.1f}M |"
         for n, c, s in rows)
     excl = "\n".join(f"| `{k}` | {v} |" for k, v in EXCLUDED.items())
+    unreleased = ""
+    if pending:
+        unreleased_rows = "\n".join(
+            f"| [{n}](data/{n}/{n}.md) | {c['pretty']} | `{c['license']}` | "
+            f"{s_['kept']:,} | {s_['tokens']/1e6:,.1f}M |"
+            for n, c, s_ in pending)
+        unreleased = (
+            "\n## Sources on main, not yet released\n"
+            f"Built and documented, but **not** part of v{VERSION} and excluded from "
+            "every total above. They join a release when a CHANGELOG entry admits "
+            "them.\n\n"
+            "| source | description | license | documents | tokens |\n"
+            "|---|---|---|---:|---:|\n"
+            f"{unreleased_rows}\n")
     phrase_frequency = ""
     phrase_path = ROOT / "artifacts" / "pattern_frequency_hf_snippet.md"
     if phrase_path.exists():
@@ -143,7 +186,7 @@ def main():
             "\n## Results\n\n"
             "### Corpus phrase frequency (normalized by tokens)\n\n"
             "Raw counts and token-normalized shares are regenerated from the "
-            "current parquet files with `src/pattern_frequency_report.py`.\n\n"
+            "current Parquet files with `src/pattern_frequency_report.py`.\n\n"
             f"{snippet}"
         )
 
@@ -206,11 +249,11 @@ family (Enevoldsen et al., [arXiv:2508.02271](https://arxiv.org/abs/2508.02271))
 
 #### v0.2.5 — current stable
 
-Added `samorzad_gov_pl`, a directly collected corpus of articles from 246
-Polish local public institutions and one central platform tenant. The
-contribution preserves the publishing institution as `author`, discovered and
-fetched article URLs in an attribution sidecar and the platform's CC-BY-SA-4.0
-basis.
+Added `samorzad_gov_pl`: **72,771 documents / 41,154,506 cl100k-proxy
+tokens** from 246 Polish local public institutions and one central platform
+tenant. The contribution preserves the publishing institution as `author`,
+discovered and fetched article URLs in an attribution sidecar and the verified
+platform-wide CC-BY-SA-4.0 basis.
 
 #### v0.2.4 — previous stable
 
@@ -315,7 +358,7 @@ files is not listed as a data contribution.
 |---|---|---|---:|---:|
 {tbl}
 | **total** | | | **{tot_doc:,}** | **{tot_tok/1e6:,.1f}M** |
-
+{unreleased}
 ## Method
 Only **human-authored** text — no synthetic, machine-translated, or auto-transcribed
 data. Gates are intentionally minimal (drop short docs, non-Polish, exact duplicates,
@@ -389,7 +432,7 @@ python3 src/make_docs.py
 ```
 {phrase_frequency}
 """
-    (ROOT / "README.md").write_text(readme)
+    write_text(ROOT / "README.md", readme.rstrip("\n") + "\n")
 
     # CHANGELOG is append-only release history. Documentation regeneration must
     # never erase earlier releases or hand-reviewed legal/release notes.
@@ -403,15 +446,19 @@ python3 src/make_docs.py
             f"{tot_doc:,} docs, {tot_tok:,} tokens (tiktoken cl100k proxy).\n\n"
         )
         changelog = changelog.replace("# Changelog\n", f"# Changelog\n\n{entry}", 1)
-        changelog_path.write_text(changelog)
+        write_text(changelog_path, changelog)
 
-    (ROOT / "LICENSE").write_text(
+    write_text(
+        ROOT / "LICENSE",
         "Polish DynaWord is released under Creative Commons Attribution-ShareAlike 4.0\n"
         "International (CC-BY-SA-4.0): https://creativecommons.org/licenses/by-sa/4.0/\n\n"
         "Per-source upstream licenses and attribution are documented in each\n"
         "data/<source>/<source>.md datasheet.\n")
 
-    print(f"docs written: README + CHANGELOG + LICENSE + {len(rows)} datasheets")
+    print(f"docs written: README + CHANGELOG + LICENSE + "
+          f"{len(rows) + len(pending)} datasheets")
+    print(f"release v{VERSION}: {len(rows)} sources"
+          + (f" | on main, unreleased: {len(pending)}" if pending else ""))
     print(f"TOTAL {tot_doc:,} docs | {tot_tok/1e9:.2f}B tok | {tot_chr/1e9:.1f}B chars")
 
 

src/sources.py

@@ -16,6 +16,8 @@ intermediate aggregator, upstream license/attribution is preserved per source.
 # be rebuilt from direct upstream/export scripts rather than blind SpeakLeash pulls.
 SOURCES = {
     "wikipedia": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "speakleash_key": "plwiki",
         "pretty": "Polish Wikipedia",
         "license": "CC-BY-SA-3.0",
@@ -28,6 +30,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wikisource": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "speakleash_key": "plwikisource",
         "pretty": "Polish Wikisource",
         "license": "CC-BY-SA-3.0",
@@ -40,6 +44,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wiktionary_examples": {
+        "added": "2026-07-19",
+        "release": None,
         "file_key": "wiktionary_examples",
         "pretty": "Polish Wiktionary usage examples",
         "provenance": "Parsed directly from the official Wikimedia dump "
@@ -58,6 +64,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "eurlex": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "speakleash_key": "eurlex_corpus",
         "pretty": "EUR-Lex (EU legal acts, Polish)",
         "license": "CC-BY-4.0",
@@ -71,6 +79,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "parliamentary": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "speakleash_key": "PPC_corpus",
         "pretty": "Polish Parliamentary Corpus (Sejm/Senat)",
         "license": "public-domain (official documents)",
@@ -84,6 +94,9 @@ SOURCES = {
         "is_ocr": False,
     },
     "sejm_api": {
+        "added": "2026-08-22",
+        "custom_datasheet": True,
+        "release": None,
         "file_key": "sejm_api",
         "pretty": "Sejm API parliamentary speeches (2023 onward)",
         "license": "public-domain (official documents)",
@@ -100,6 +113,7 @@ SOURCES = {
         "is_ocr": False,
     },
     "sejm_interpellations": {
+        "release": None,
         "file_key": "sejm_interpellations",
         "pretty": "Sejm interpellations and written questions (terms 7–10)",
         "license": "public-domain (official documents)",
@@ -119,6 +133,8 @@ SOURCES = {
         "custom_datasheet": True,
     },
     "parlamint_pl": {
+        "added": "2026-07-19",
+        "release": None,
         "file_key": "parlamint_pl",
         "pretty": "ParlaMint-PL (parliamentary debates, 2015-2022)",
         "license": "CC-BY-4.0",
@@ -136,6 +152,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wolne_lektury": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "speakleash_key": "wolne_lektury_corpus",
         "pretty": "Wolne Lektury (school readings)",
         "license": "CC-BY-SA-4.0 / Wolna Sztuka 1.3",
@@ -148,6 +166,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wikinews": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "file_key": "plwikinews",  # fetched from Wikimedia dumps, NOT via SpeakLeash
         "pretty": "Polish Wikinews",
         "license": "CC-BY-2.5",
@@ -163,6 +183,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wikivoyage": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "file_key": "plwikivoyage",
         "pretty": "Polish Wikivoyage (travel guides)",
         "license": "CC-BY-SA-3.0",
@@ -178,6 +200,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wikibooks": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "file_key": "plwikibooks",
         "pretty": "Polish Wikibooks (open textbooks)",
         "license": "CC-BY-SA-3.0",
@@ -193,6 +217,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "wikiquote": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "file_key": "plwikiquote",
         "pretty": "Polish Wikiquote (quotations)",
         "license": "CC-BY-SA-3.0",
@@ -208,6 +234,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "eltec_pol": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "file_key": "eltec_pol",
         "pretty": "ELTeC-pol (European Literary Text Collection, Polish)",
         "license": "CC-BY-4.0",
@@ -223,6 +251,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "dziennik_ustaw": {
+        "added": "2026-07-02",
+        "release": "0.2.0",
         "file_key": "dziennik_ustaw",  # fetched directly, NOT via SpeakLeash
         "pretty": "Dziennik Ustaw + Monitor Polski (Polish primary legislation)",
         "license": "public-domain (official documents)",
@@ -240,6 +270,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "govpl": {
+        "added": "2026-07-19",
+        "release": "0.2.2",
         "file_key": "govpl",  # fetched directly from gov.pl, NOT via SpeakLeash
         "pretty": "gov.pl — Polish government press releases",
         "license": "CC-BY-SA-4.0",
@@ -262,6 +294,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "samorzad_gov_pl": {
+        "added": "2026-08-14",
+        "release": "0.2.5",
         "file_key": "samorzad_gov_pl",
         "pretty": "samorzad.gov.pl — Polish public-sector institutions",
         "license": "CC-BY-SA-4.0",
@@ -285,6 +319,9 @@ SOURCES = {
         "custom_datasheet": True,
     },
     "biblioteka_nauki": {
+        "added": "2026-06-27",
+        "custom_datasheet": True,
+        "release": "0.2.4",
         "file_key": "biblioteka_nauki_pl_corpus",
         "pretty": "Biblioteka Nauki",
         "license": "per-record upstream license",
@@ -303,6 +340,7 @@ SOURCES = {
         "is_ocr": False,
     },
     "europeana": {
+        "release": None,
         # Direct rebuild output from fetch_europeana.py, never the raw
         # SpeakLeash Europeana aggregate.
         "file_key": "europeana",
@@ -328,6 +366,9 @@ SOURCES = {
                       "legal review confirms their US status.",
     },
     "nkjp1m": {
+        "added": "2026-06-15",
+        "custom_datasheet": True,
+        "release": "0.2.3",
         "file_key": "nkjp1m",
         "pretty": "The manually annotated 1-million word subcorpus of the National Corpus of Polish",
         "license": "CC-BY",
@@ -340,6 +381,8 @@ SOURCES = {
         "is_ocr": False,
     },
     "european_hplt_v3_pl": {
+        "added": "2026-07-14",
+        "release": "0.2.3",
         "file_key": "european_hplt_v3_pl",
         "pretty": "HPLT v3.0 Polish (web, top WDS bins 10-5)",
         "license": "CC0-1.0",
@@ -371,6 +414,8 @@ SOURCES = {
         "custom_datasheet": True,
     },
     "1000_novels": {
+        "added": "2026-07-02",
+        "release": "0.2.1",
         "speakleash_key": "1000_novels_corpus_CLARIN-PL",
         "pretty": "1000 Novels Corpus (CLARIN-PL)",
         "license": "CC-BY-4.0",
@@ -388,6 +433,8 @@ SOURCES = {
         "custom_datasheet": True,
     },
     "global_voices": {
+        "added": "2026-07-15",
+        "release": "0.2.3",
         # Data contributed directly (PR #7): no in-repo fetch/build script and a
         # hand-authored datasheet. Registered for the source table + release totals;
         # the datasheet stays contributor-owned via custom_datasheet.
@@ -407,6 +454,7 @@ SOURCES = {
     },
 }
 
+# Fallback stamp for a source with no recorded "added" date of its own.
 ADDED = "2026-07-19"
 EXCLUDED = {
     "open_subtitles_corpus": "Derivative of copyrighted film/TV dialogue; "

src/test_verify_paper_claims.py

@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+"""Self-check for src/verify_paper_claims.py against a synthetic mini-repo."""
+
+import json
+import shutil
+import sys
+import tempfile
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import verify_paper_claims as vpc  # noqa: E402
+
+TEX = r"""
+\begin{figure}
+  \addplot[thick] coordinates {(v0.1.0,1.00)(v0.1.1,3.00)};
+  \caption{grew from 1.00 billion tokens and 1 sources (v0.1.0).}
+\end{figure}
+\begin{abstract}
+Release v0.1.1 comprises \num{300} documents and \num{3.00}~billion tokens
+from 2 sources.
+\end{abstract}
+\begin{table}
+\begin{tabular}{lrr}
+Version & Tokens (billions) & Sources \\
+v0.1.0 & 1.00 & 1 \\
+v0.1.1 & 3.00 & 2 \\
+\end{tabular}
+\label{tab:s04-wersje}
+\end{table}
+\begin{table}
+\label{tab:s05-zrodla}
+\begin{tabular}{llr}
+Source & License & Tokens [M] \\
+\texttt{alpha}      & CC0       & 2000.0 \\
+\texttt{beta\_two}  & CC-BY-SA-4.0 & 1000.0 \\
+\textbf{Total} & CC-BY-SA-4.0 & \textbf{$\approx$3000} \\
+\end{tabular}
+\end{table}
+\begin{table}
+\label{tab:s10-licencje}
+\begin{tabular}{llr}
+License family & Representative sources & Tokens [billions] \\
+CC0            & alpha    & $2.00$ \\
+CC-BY-SA       & beta two & $1.00$ \\
+Total          &          & $3.00$ \\
+\end{tabular}
+\end{table}
+This work is pinned to commit \texttt{deadbeef}.
+"""
+
+CHANGELOG = """# Changelog
+
+## v0.1.1 (2026-01-02)
+
+- Added `beta_two`.
+- Totals: 2 sources, 300 docs, 3.00B tokens.
+"""
+
+README = "| `v0.1.0` | previous stable | 100 | 1.00B | Baseline from 1 open/official sources. |\n"
+
+GROWTH = """VERSIONS = ["v0.1.0", "v0.1.1"]
+DOCUMENTS = [100, 300]
+TOKENS = [1_000_000_000, 3_000_000_000]
+"""
+
+STATS = {"alpha": (200, 2_000_000_000), "beta_two": (100, 1_000_000_000)}
+
+SOURCES = """SOURCES = {
+    "alpha": {"license": "CC0-1.0", "license_spdx": "CC0-1.0", "release": "0.1.0"},
+    "beta_two": {"license": "CC-BY-SA-4.0", "license_spdx": "CC-BY-SA-4.0", "release": "0.1.1"},
+}
+"""
+
+
+def registry(root: Path) -> dict:
+    return vpc.load_registry(root)
+
+
+def edit(path: Path, old: str, new: str) -> None:
+    text = path.read_text()
+    assert old in text, f"fixture no longer contains {old!r}"
+    path.write_text(text.replace(old, new))
+
+
+def build(root: Path) -> None:
+    (root / "src").mkdir(parents=True)
+    (root / "CHANGELOG.md").write_text(CHANGELOG)
+    (root / "README.md").write_text(README)
+    (root / "src" / "plot_version_growth.py").write_text(GROWTH)
+    (root / "src" / "sources.py").write_text(SOURCES)
+    for name, (docs, tokens) in STATS.items():
+        directory = root / "data" / name
+        directory.mkdir(parents=True)
+        (directory / f"{name}.stats.json").write_text(
+            json.dumps({"kept": docs, "tokens": tokens, "license": "CC0"})
+        )
+        (directory / f"{name}.md").write_text(
+            f"# {name}\n- **License:** `{registry(root)[name]['license']}`\n"
+        )
+    (root / "paper.tex").write_text(TEX)
+
+
+def run(root: Path) -> int:
+    return vpc.main(["--tex", str(root / "paper.tex"), "--root", str(root), "--ref", "worktree"])
+
+
+def scenario(mutate=None) -> int:
+    root = Path(tempfile.mkdtemp())
+    try:
+        build(root)
+        if mutate:
+            mutate(root)
+        return run(root)
+    finally:
+        shutil.rmtree(root)
+
+
+def drop_token_digit(root: Path) -> None:
+    path = root / "paper.tex"
+    path.write_text(path.read_text().replace("2000.0", "2500.0"))
+
+
+def wrong_source_count(root: Path) -> None:
+    path = root / "paper.tex"
+    path.write_text(path.read_text().replace("v0.1.1 & 3.00 & 2", "v0.1.1 & 3.00 & ---"))
+
+
+def add_source(root: Path, release: str | None) -> None:
+    directory = root / "data" / "gamma"
+    directory.mkdir(parents=True)
+    (directory / "gamma.stats.json").write_text(json.dumps({"kept": 1, "tokens": 1, "license": "CC0"}))
+    (directory / "gamma.md").write_text("# gamma\n- **License:** `CC0-1.0`\n")
+    edit(root / "src" / "sources.py", "}\n",
+         f'    "gamma": {{"license": "CC0-1.0", "release": {release!r}}},\n}}\n')
+
+
+def unlisted_source(root: Path) -> None:
+    add_source(root, "0.1.1")
+
+
+def unreleased_source(root: Path) -> None:
+    add_source(root, None)
+
+
+def wrong_license(root: Path) -> None:
+    edit(root / "paper.tex", "CC-BY-SA-4.0 & 1000.0", "CC-BY-4.0 & 1000.0")
+
+
+def datasheet_license_drift(root: Path) -> None:
+    edit(root / "data" / "alpha" / "alpha.md", "CC0-1.0", "CC-BY-4.0")
+
+
+def undefined_footnote(root: Path) -> None:
+    edit(root / "paper.tex", "& CC0       &", r"& CC0$^{\dagger}$ &")
+
+
+def family_value_drift(root: Path) -> None:
+    edit(root / "paper.tex", "& beta two & $1.00$", "& beta two & $1.50$")
+
+
+def incomplete_partition(root: Path) -> None:
+    edit(root / "paper.tex", "CC-BY-SA       & beta two & $1.00$ \\\\\n", "")
+
+
+def unfootnoted_dual_license(root: Path) -> None:
+    """alpha's content is CC0 but its platform redistributes under CC-BY-4.0."""
+    edit(root / "src" / "sources.py",
+         '"alpha": {"license": "CC0-1.0", "license_spdx": "CC0-1.0"',
+         '"alpha": {"license": "CC0-1.0", "license_spdx": "CC-BY-4.0"')
+
+
+def stale_growth_series(root: Path) -> None:
+    path = root / "src" / "plot_version_growth.py"
+    path.write_text(path.read_text().replace("3_000_000_000", "3_100_000_000"))
+
+
+if __name__ == "__main__":
+    assert scenario() == 0, "consistent fixture must pass"
+    assert scenario(drop_token_digit) == 1, "per-source token drift must fail"
+    assert scenario(wrong_source_count) == 1, "carried-over '---' source count must fail"
+    assert scenario(unlisted_source) == 1, "source present in repo but not in Table 3 must fail"
+    assert scenario(stale_growth_series) == 1, "growth series vs composition drift must fail"
+    assert scenario(unreleased_source) == 0, "a source with release=None is not part of the release"
+    assert scenario(wrong_license) == 1, "license in Table 3 vs registry must fail"
+    assert scenario(datasheet_license_drift) == 1, "datasheet vs registry license drift must fail"
+    assert scenario(undefined_footnote) == 1, "a footnote marker with no definition must fail"
+    assert scenario(unfootnoted_dual_license) == 1, "an unfootnoted dual license must fail"
+    assert scenario(family_value_drift) == 1, "a wrong license-family subtotal must fail"
+    assert scenario(incomplete_partition) == 1, "a license family with no row must fail"
+    print("ok")

src/verify_paper_claims.py

@@ -0,0 +1,637 @@
+#!/usr/bin/env python3
+"""Verify the paper's numeric claims against repository ground truth.
+
+Two checks, matching the two things the paper asserts numerically:
+
+  1. Release trajectory (Table `tab:s04-wersje` + the hero figure):
+     tokens and source counts for v0.2.0 -> v0.2.5.
+     Ground truth: `src/plot_version_growth.py` literals (exact docs/tokens)
+     and CHANGELOG.md / README.md (source counts per release).
+
+  2. Corpus composition (Table `tab:s05-zrodla` + the inline totals):
+     per-source token counts, the source set, and the corpus totals.
+     Ground truth: `data/*/*.stats.json` at a git ref.
+
+Usage:
+    python src/verify_paper_claims.py                    # ref = pin found in the .tex
+    python src/verify_paper_claims.py --ref worktree     # current working tree
+    python src/verify_paper_claims.py --tex paper.tex --ref HEAD
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import json
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+
+# Printed precision in the paper sets the comparison tolerance.
+TOL_BILLIONS = 0.005  # Table 2 / hero figure: 2 decimal places
+TOL_MILLIONS = 0.05  # Table 3: 1 decimal place
+
+OK, FAIL, WARN = "OK", "FAIL", "WARN"
+
+MARKER = re.compile(r"\$\^\{([^}]*)\}\$")
+
+
+class Report:
+    def __init__(self) -> None:
+        self.rows: list[tuple[str, str, str]] = []
+
+    def add(self, status: str, label: str, detail: str = "") -> None:
+        self.rows.append((status, label, detail))
+
+    def num(self, label: str, claimed, actual, tol: float = 0.0) -> None:
+        if claimed is None:
+            self.add(WARN, label, "not found in .tex")
+        elif abs(claimed - actual) <= tol:
+            self.add(OK, label, f"{claimed} == {actual}")
+        else:
+            self.add(FAIL, label, f"paper {claimed} != repo {actual}")
+
+    def section(self, title: str) -> None:
+        self.rows.append(("", title, ""))
+
+    def render(self) -> int:
+        width = max(len(label) for _, label, _ in self.rows) + 2
+        for status, label, detail in self.rows:
+            if not status:
+                print(f"\n=== {label} ===")
+            else:
+                print(f"  [{status:<4}] {label:<{width}} {detail}")
+        failures = sum(1 for s, _, _ in self.rows if s == FAIL)
+        warnings = sum(1 for s, _, _ in self.rows if s == WARN)
+        print(f"\n{failures} failure(s), {warnings} warning(s)")
+        return 1 if failures else 0
+
+
+# --------------------------------------------------------------------------
+# .tex parsing
+# --------------------------------------------------------------------------
+
+def _unescape(name: str) -> str:
+    return name.replace("\\_", "_").replace("\\&", "&")
+
+
+def _table_body(tex: str, label: str) -> str:
+    """Return the body of the table environment carrying \\label{label}."""
+    for block in re.findall(r"\\begin\{table\}.*?\\end\{table\}", tex, re.S):
+        if f"\\label{{{label}}}" in block:
+            return block
+    raise LookupError(f"no table with \\label{{{label}}} in the .tex")
+
+
+def parse_hero(tex: str) -> dict[str, float]:
+    """Version -> tokens (billions) from the pgfplots hero figure coordinates."""
+    coords = re.search(r"\\addplot\[[^\]]*\]\s*coordinates\s*\{(.*?)\}", tex, re.S)
+    if not coords:
+        return {}
+    return {v: float(t) for v, t in re.findall(r"\((v[\d.]+),\s*([\d.]+)\)", coords.group(1))}
+
+
+def parse_trajectory_table(tex: str) -> dict[str, tuple[float, str]]:
+    """Version -> (tokens in billions, source-count cell verbatim)."""
+    body = _table_body(tex, "tab:s04-wersje")
+    rows = re.findall(r"^\s*(v[\d.]+)\s*&\s*([\d.]+)\s*&\s*(.+?)\s*\\\\", body, re.M)
+    return {v: (float(tok), src.strip()) for v, tok, src in rows}
+
+
+def parse_source_table(tex: str) -> tuple[dict[str, float], float | None]:
+    """(source name -> tokens in millions, declared total in millions)."""
+    body = _table_body(tex, "tab:s05-zrodla")
+    sources = {
+        _unescape(name): float(tok)
+        for name, tok in re.findall(
+            r"\\texttt\{([^}]+)\}\s*&.*?&\s*([\d.]+)\s*\\\\", body
+        )
+    }
+    total_row = re.search(r"\\textbf\{Total\}.*?&.*?&\s*(.+?)\s*\\\\", body, re.S)
+    total = None
+    if total_row:
+        digits = re.search(r"([\d.]+)", total_row.group(1))
+        if digits:
+            total = float(digits.group(1))
+    return sources, total
+
+
+def parse_license_rows(tex: str) -> dict[str, tuple[str, set[str]]]:
+    """Table 3 source rows -> (license cell without its markers, marker names)."""
+    body = _table_body(tex, "tab:s05-zrodla").split(r"\begin{tabular}")[-1]
+    rows = {}
+    for name, cell in re.findall(r"\\texttt\{([^}]+)\}\s*&(.*?)&\s*[\d.]+\s*\\\\", body):
+        rows[_unescape(name)] = (MARKER.sub("", cell).strip(), _markers(cell))
+    return rows
+
+
+def parse_footnote_defs(tex: str) -> set[str]:
+    """Markers the Table 3 caption actually explains (a definition is followed by ~)."""
+    caption = _table_body(tex, "tab:s05-zrodla").split(r"\begin{tabular}")[0]
+    return {m for cell in re.findall(r"(\$\^\{[^}]*\}\$)~", caption) for m in _markers(cell)}
+
+
+def parse_total_license(tex: str) -> str | None:
+    body = _table_body(tex, "tab:s05-zrodla")
+    row = re.search(r"\\textbf\{Total\}\s*&\s*(.*?)\s*&", body)
+    return MARKER.sub("", row.group(1)).strip() if row else None
+
+
+def _markers(cell: str) -> set[str]:
+    return {m for group in MARKER.findall(cell) for m in re.findall(r"\\([a-zA-Z]+)", group)}
+
+
+def parse_family_table(tex: str) -> tuple[dict[str, float], float | None]:
+    """Table 5 -> (license family -> tokens in billions, declared total)."""
+    body = _table_body(tex, "tab:s10-licencje").split(r"\begin{tabular}")[-1]
+    families, total = {}, None
+    for label, value in re.findall(r"^\s*([^&\\]+?)\s*&[^&]*&\s*\$?([\d.]+)\$?\s*\\\\",
+                                   body, re.M):
+        if label.strip().lower() == "total":
+            total = float(value)
+        elif license_family(label):
+            families[license_family(label)] = float(value)
+    return families, total
+
+
+def license_family(text: str) -> str | None:
+    """The Table 5 partition: like norm_license, but CC0 and public domain split.
+
+    norm_license folds them together because the source table prints either for
+    the same source; Table 5 reports them as separate families, so a source's
+    own wording decides which side it lands on.
+    """
+    t = re.sub(r"[`_]", " ", text).upper()
+    if re.search(r"PER-RECORD|PER-DOCUMENT|MIXED", t):
+        return "Per-record (mixed)"
+    if re.search(r"PUBLIC[- ]DOMAIN", t):
+        return "Public domain"
+    if "CC0" in t:
+        return "CC0"
+    if re.search(r"CC[- ]BY[- ]SA", t):
+        return "CC-BY-SA"
+    if re.search(r"CC[- ]BY", t):
+        return "CC-BY"
+    return None
+
+
+def parse_inline_claims(tex: str, current: str | None = None) -> dict:
+    """Claims the prose/captions make about the *current release* totals.
+
+    A claim is skipped when its surrounding sentence scopes it to a different
+    release or to a partition subtotal, so only whole-release assertions are
+    compared against the current release's ground truth.
+    """
+    CONTEXT = 160
+    SUBTOTAL = re.compile(r"share-alike|partitions?\b")
+    VERSION = re.compile(r"v\d+\.\d+\.\d+")
+
+    def sweep(pattern: str, cast):
+        out = set()
+        for match in re.finditer(pattern, tex):
+            window = tex[max(0, match.start() - CONTEXT):match.end() + CONTEXT]
+            if SUBTOTAL.search(window):
+                continue
+            if any(v != current for v in VERSION.findall(window)):
+                continue
+            raw = next(g for g in match.groups() if g)
+            out.add(cast(raw))
+        return out
+
+    docs = sweep(
+        r"(?:\\num\{(\d+)\}|(\d[\d\\,]*\d))\s*~?\s*documents",
+        lambda raw: int(re.sub(r"[^\d]", "", raw)),
+    )
+    # ponytail: floor keeps small unrelated "N documents" counts (held-out set
+    # sizes) out; scope by section if the paper grows other six-figure claims.
+    docs = {value for value in docs if value >= 100_000}
+
+    tokens = sweep(
+        r"(?:\\num\{([\d.]+)\}|\$([\d.]+)\$|([\d.]+))\s*~?\s*billion\s+tokens",
+        float,
+    )
+    sources = sweep(r"\b(\d+)\s+sources\b", int)
+
+    pin = re.search(r"commit\s+\\texttt\{([0-9a-f]{7,40})\}", tex)
+    return {
+        "documents": docs,
+        "tokens_billions": tokens,
+        "sources": sources,
+        "pin": pin.group(1) if pin else None,
+    }
+
+
+# --------------------------------------------------------------------------
+# Ground truth
+# --------------------------------------------------------------------------
+
+def load_growth_series(root: Path) -> dict[str, tuple[int, int]]:
+    """Version -> (documents, tokens) from src/plot_version_growth.py literals."""
+    path = root / "src" / "plot_version_growth.py"
+    tree = ast.parse(path.read_text(encoding="utf-8"))
+    found: dict[str, list] = {}
+    for node in tree.body:
+        if isinstance(node, ast.Assign) and len(node.targets) == 1:
+            target = node.targets[0]
+            if isinstance(target, ast.Name) and target.id in {"VERSIONS", "DOCUMENTS", "TOKENS"}:
+                found[target.id] = ast.literal_eval(node.value)
+    missing = {"VERSIONS", "DOCUMENTS", "TOKENS"} - found.keys()
+    if missing:
+        raise LookupError(f"{path} is missing {sorted(missing)}")
+    return dict(zip(found["VERSIONS"], zip(found["DOCUMENTS"], found["TOKENS"])))
+
+
+def load_registry(root: Path) -> dict[str, dict]:
+    """The SOURCES table from src/sources.py, read without importing the module.
+
+    Always read from the working tree: the registry defines what a release *is*,
+    independently of the ref the parquet stats are read from.
+    """
+    path = root / "src" / "sources.py"
+    if not path.exists():
+        return {}
+    for node in ast.parse(path.read_text(encoding="utf-8")).body:
+        if (isinstance(node, ast.Assign) and len(node.targets) == 1
+                and getattr(node.targets[0], "id", None) == "SOURCES"):
+            return ast.literal_eval(node.value)
+    return {}
+
+
+def load_release_map(root: Path) -> dict[str, str | None]:
+    """Source name -> the release it entered.
+
+    Empty when the registry is absent or predates the `release` field, in which
+    case the CHANGELOG prose stays the only ground truth.
+    """
+    sources = load_registry(root)
+    if not any("release" in cfg for cfg in sources.values()):
+        return {}
+    return {name: cfg.get("release") for name, cfg in sources.items()}
+
+
+def released_at(releases: dict[str, str | None], version: str) -> set[str]:
+    """Sources belonging to `version`, i.e. admitted at or before it."""
+    ceiling = _version_key(version)
+    return {
+        name for name, release in releases.items()
+        if release and _version_key(release) <= ceiling
+    }
+
+
+def _version_key(version: str) -> tuple[int, ...]:
+    return tuple(int(part) for part in version.lstrip("v").split("."))
+
+
+def _changelog_sections(root: Path) -> dict[str, str]:
+    text = (root / "CHANGELOG.md").read_text(encoding="utf-8")
+    parts = re.split(r"^## (v[\d.]+[\w.-]*)", text, flags=re.M)
+    return dict(zip(parts[1::2], parts[2::2]))
+
+
+def load_declared_sources(
+    root: Path, base_version: str | None = None
+) -> tuple[dict[str, int | None], dict[str, list[str]]]:
+    """Version -> declared source count (None when unrecorded), and sources added."""
+    sections = _changelog_sections(root)
+    counts: dict[str, int | None] = {}
+    added: dict[str, list[str]] = {}
+    for version, body in sections.items():
+        match = re.search(r"\b(\d+)[-\s]sources?\b", body)
+        counts[version] = int(match.group(1)) if match else None
+        added[version] = re.findall(
+            r"Added (?:the )?(?:community[- ]contributed |community contribution |reviewed )?`([\w.]+)`",
+            body,
+        )
+    # The first release predates the CHANGELOG; the README release table records it.
+    if base_version and counts.get(base_version) is None:
+        readme = (root / "README.md").read_text(encoding="utf-8")
+        base = re.search(
+            rf"`{re.escape(base_version)}`.*?(\d+) open/official sources", readme, re.S
+        )
+        if base:
+            counts[base_version] = int(base.group(1))
+            added.setdefault(base_version, [])
+    return counts, added
+
+
+def load_stats(root: Path, ref: str) -> dict[str, dict]:
+    """Source name -> parquet stats, read from a git ref or the working tree."""
+    stats: dict[str, dict] = {}
+    if ref == "worktree":
+        paths = sorted(root.glob("data/*/*.stats.json"))
+        for path in paths:
+            stats[path.name.removesuffix(".stats.json")] = json.loads(
+                path.read_text(encoding="utf-8")
+            )
+        return stats
+
+    listing = subprocess.run(
+        ["git", "-C", str(root), "ls-tree", "-r", "--name-only", ref, "--", "data/"],
+        capture_output=True, text=True, check=True,
+    ).stdout.splitlines()
+    for rel in sorted(p for p in listing if p.endswith(".stats.json")):
+        blob = subprocess.run(
+            ["git", "-C", str(root), "show", f"{ref}:{rel}"],
+            capture_output=True, text=True, check=True,
+        ).stdout
+        stats[Path(rel).name.removesuffix(".stats.json")] = json.loads(blob)
+    return stats
+
+
+# --------------------------------------------------------------------------
+# Checks
+# --------------------------------------------------------------------------
+
+def check_trajectory(report: Report, tex: str, root: Path) -> None:
+    report.section("Check 1 - release trajectory v0.2.0 -> v0.2.5 (Table 2 + hero figure)")
+
+    table = parse_trajectory_table(tex)
+    hero = parse_hero(tex)
+    series = load_growth_series(root)
+    declared, added = load_declared_sources(root, base_version=sorted(series)[0])
+    releases = load_release_map(root)
+    if releases:
+        for version in sorted(series):
+            counted = len(released_at(releases, version))
+            if declared.get(version) is None:
+                declared[version] = counted
+            elif declared[version] != counted:
+                report.add(FAIL, f"{version} registry vs CHANGELOG",
+                           f"src/sources.py counts {counted}, CHANGELOG says {declared[version]}")
+
+    missing = sorted(set(series) - set(table))
+    extra = sorted(set(table) - set(series))
+    if missing or extra:
+        report.add(FAIL, "table covers every release",
+                   f"missing {missing}, unexpected {extra}")
+    else:
+        report.add(OK, "table covers every release", f"{len(table)} rows")
+
+    for version in sorted(series):
+        docs, tokens = series[version]
+        actual_b = tokens / 1e9
+        claimed_b = table.get(version, (None, None))[0]
+        report.num(f"{version} tokens (B, Table 2)", claimed_b, round(actual_b, 6), TOL_BILLIONS)
+        report.num(f"{version} tokens (B, hero figure)", hero.get(version), round(actual_b, 6), TOL_BILLIONS)
+
+    # Source counts. "---" in the table means "unchanged from the previous release".
+    previous: int | None = None
+    for version in sorted(series):
+        cell = table.get(version, (None, "?"))[1]
+        if cell == "---":
+            claimed = previous
+            shown = f"--- (carries {previous})"
+        else:
+            digits = re.search(r"\d+", cell or "")
+            claimed = int(digits.group()) if digits else None
+            shown = str(claimed)
+        previous = claimed
+
+        truth = declared.get(version)
+        label = f"{version} source count"
+        if truth is None:
+            report.add(WARN, label, f"paper {shown}; no count recorded in CHANGELOG/README")
+        elif claimed == truth:
+            report.add(OK, label, f"{shown} == {truth}")
+        else:
+            report.add(FAIL, label, f"paper {shown} != repo {truth}")
+
+    if releases:
+        return  # the registry already resolves every release exactly
+
+    # Back-fill unrecorded releases from the next release's "Added `source`" bullets.
+    versions = sorted(series)
+    for index, version in enumerate(versions[:-1]):
+        if declared.get(version) is not None:
+            continue
+        following = versions[index + 1]
+        if declared.get(following) is None:
+            continue
+        inferred = declared[following] - len(added.get(following, []))
+        report.add(
+            WARN,
+            f"{version} source count (derived)",
+            f"{following} declares {declared[following]} and adds "
+            f"{len(added.get(following, []))} -> {version} = {inferred}",
+        )
+
+
+def check_composition(report: Report, tex: str, root: Path, ref: str) -> None:
+    report.section(f"Check 2 - corpus composition and totals (Table 3, ref={ref})")
+
+    table, table_total = parse_source_table(tex)
+    current = sorted(load_growth_series(root))[-1]
+    claims = parse_inline_claims(tex, current=current)
+    stats = load_stats(root, ref)
+    releases = load_release_map(root)
+    in_release = released_at(releases, current) if releases else set(stats)
+
+    absent = sorted(set(table) - set(stats))
+    unlisted = sorted((set(stats) & in_release) - set(table))
+    unreleased = sorted(set(stats) - in_release)
+    listed_but_unreleased = sorted(set(table) & set(unreleased))
+    if absent:
+        report.add(FAIL, "every listed source exists at ref", f"missing from repo: {absent}")
+    else:
+        report.add(OK, "every listed source exists at ref", f"{len(table)} sources")
+    if unlisted:
+        report.add(FAIL, f"every {current} source is in Table 3",
+                   f"released but missing from Table 3: {unlisted}")
+    else:
+        report.add(OK, f"every {current} source is in Table 3", "")
+    if listed_but_unreleased:
+        report.add(FAIL, "Table 3 lists only released sources",
+                   f"not in any release: {listed_but_unreleased}")
+    elif unreleased:
+        report.add(OK, "Table 3 lists only released sources",
+                   f"correctly excludes {unreleased}")
+
+    listed = [name for name in table if name in stats]
+    for name in sorted(listed, key=lambda n: -stats[n]["tokens"]):
+        report.num(f"{name} tokens (M)", table[name], round(stats[name]["tokens"] / 1e6, 6), TOL_MILLIONS)
+
+    sum_tokens = sum(stats[name]["tokens"] for name in listed)
+    sum_docs = sum(stats[name]["kept"] for name in listed)
+
+    report.num("Table 3 rows sum to its Total row (M)",
+               round(sum(table[n] for n in listed), 6),
+               round(sum_tokens / 1e6, 6), TOL_MILLIONS * len(listed))
+    report.num("Table 3 Total row (M)", table_total, round(sum_tokens / 1e6, 6), 1.0)
+
+    for value in sorted(claims["documents"]):
+        report.num(f"prose claim: {value} documents", value, sum_docs)
+    for value in sorted(claims["tokens_billions"]):
+        report.num(f"prose claim: {value}B tokens", value, round(sum_tokens / 1e9, 6), TOL_BILLIONS)
+    for value in sorted(claims["sources"]):
+        report.num(f"prose claim: {value} sources", value, len(listed))
+
+    if unreleased:
+        report.add(
+            WARN, "totals over ALL sources at ref",
+            f"{sum(s['kept'] for s in stats.values())} docs / "
+            f"{sum(s['tokens'] for s in stats.values()) / 1e9:.3f}B tokens "
+            f"across {len(stats)} sources",
+        )
+
+    # Anchor: the trajectory series' last point must equal the composition sum.
+    series = load_growth_series(root)
+    last = sorted(series)[-1]
+    report.num(f"{last} tokens in growth series == Table 3 sum",
+               series[last][1], sum_tokens)
+    report.num(f"{last} documents in growth series == Table 3 sum",
+               series[last][0], sum_docs)
+
+
+def norm_license(text: str) -> tuple[str, str | None]:
+    """Canonical (family, version) so paper, registry and datasheet can be compared.
+
+    Public domain and CC0 are one family: SPDX spells "official documents are
+    outside copyright" as CC0-1.0, and the paper prints both as PD. A version of
+    None means the text states no version, and compares equal to any version --
+    the paper abbreviates nkjp1m's CC-BY-4.0 to "CC-BY" on purpose.
+    """
+    t = re.sub(r"[`_]", " ", text).upper()
+    if re.search(r"PER-RECORD|PER-DOCUMENT|MIXED", t):
+        return ("MIXED", None)
+    if re.search(r"CC0|PUBLIC[- ]DOMAIN|\bPD\b", t):
+        return ("PD/CC0", None)
+    match = re.search(r"CC[- ]BY(?:[- ](SA))?(?:[- ](\d+\.\d+))?", t)
+    if match:
+        return ("CC-BY-SA" if match.group(1) else "CC-BY", match.group(2))
+    return (t.strip(), None)
+
+
+def compatible(a: tuple[str, str | None], b: tuple[str, str | None]) -> bool:
+    """Same family, and same version unless one side states none."""
+    return a[0] == b[0] and (a[1] is None or b[1] is None or a[1] == b[1])
+
+
+def load_datasheet_license(root: Path, name: str) -> str | None:
+    """The datasheet's declared License line (working tree, not the pinned ref)."""
+    path = root / "data" / name / f"{name}.md"
+    if not path.exists():
+        return None
+    match = re.search(r"\*\*License:\*\*\s*(.+)", path.read_text(encoding="utf-8"))
+    return match.group(1).strip() if match else None
+
+
+def check_families(report: Report, tex: str, root: Path, ref: str) -> None:
+    """Table 5 must be a complete partition of Table 3 by license family."""
+    paper, paper_total = parse_family_table(tex)
+    registry, stats = load_registry(root), load_stats(root, ref)
+    releases = load_release_map(root)
+    if not (paper and registry and releases):
+        report.add(WARN, "license family table", "no family table or no registry to partition")
+        return
+
+    repo: dict[str, int] = {}
+    unclassified = []
+    for name in released_at(releases, sorted(load_growth_series(root))[-1]):
+        cfg = registry[name]
+        # The row prints the content's own status; SPDX carries the CC0 wording.
+        family = license_family(str(cfg.get("license", ""))) or license_family(str(cfg.get("license_spdx") or ""))
+        if family is None:
+            unclassified.append(name)
+        else:
+            repo[family] = repo.get(family, 0) + stats[name]["tokens"]
+
+    if unclassified:
+        report.add(FAIL, "every source has a license family", f"unclassifiable: {sorted(unclassified)}")
+    if set(paper) != set(repo):
+        report.add(FAIL, "Table 5 families match the corpus",
+                   f"paper {sorted(paper)} != repo {sorted(repo)}")
+    else:
+        report.add(OK, "Table 5 families match the corpus", f"{len(repo)} families, no source left over")
+
+    for family in sorted(repo, key=lambda f: -repo[f]):
+        if family in paper:
+            report.num(f"{family} tokens (B)", paper[family], round(repo[family] / 1e9, 6), TOL_BILLIONS)
+    report.num("Table 5 Total row (B)", paper_total, round(sum(repo.values()) / 1e9, 6), TOL_BILLIONS)
+
+
+def check_licenses(report: Report, tex: str, root: Path) -> None:
+    report.section("Check 3 - licenses, footnotes and license families")
+
+    rows = parse_license_rows(tex)
+    registry = load_registry(root)
+    if not registry:
+        report.add(WARN, "license audit", "src/sources.py carries no registry to compare against")
+        return
+
+    for name in sorted(rows, key=lambda n: n.lower()):
+        paper, markers = rows[name]
+        cfg = registry.get(name)
+        if cfg is None:
+            report.add(FAIL, f"{name} in registry", "listed in Table 3 but absent from src/sources.py")
+            continue
+
+        declared = norm_license(str(cfg.get("license", "")))
+        spdx = norm_license(str(cfg.get("license_spdx") or cfg.get("license", "")))
+        printed = norm_license(paper)
+
+        if compatible(printed, declared) or compatible(printed, spdx):
+            report.add(OK, f"{name} license", f"{paper} == {cfg.get('license')}")
+        else:
+            report.add(FAIL, f"{name} license",
+                       f"paper {paper!r} != registry {cfg.get('license')!r} / {cfg.get('license_spdx')!r}")
+
+        sheet = load_datasheet_license(root, name)
+        if sheet is None:
+            report.add(WARN, f"{name} datasheet license", "no **License:** line in the datasheet")
+        elif not compatible(norm_license(sheet), declared):
+            report.add(FAIL, f"{name} datasheet license",
+                       f"datasheet {sheet!r} != registry {cfg.get('license')!r}")
+
+        # A source whose content license differs from its redistribution license
+        # states only half the story in a one-line cell; the footnote carries the rest.
+        if not compatible(declared, spdx) and not markers:
+            report.add(FAIL, f"{name} dual license is footnoted",
+                       f"content {cfg.get('license')!r} but redistributed as "
+                       f"{cfg.get('license_spdx')!r}; the row prints only {paper!r}")
+
+    used = {m for _, markers in rows.values() for m in markers}
+    defined = parse_footnote_defs(tex)
+    if used - defined:
+        report.add(FAIL, "every footnote marker is defined", f"used but not explained: {sorted(used - defined)}")
+    elif defined - used:
+        report.add(FAIL, "every footnote is used", f"explained but on no row: {sorted(defined - used)}")
+    else:
+        report.add(OK, "footnote markers", f"{len(used)} defined and used: {sorted(used)}")
+
+    total = parse_total_license(tex)
+    strongest = [n for n, (lic, _) in rows.items() if norm_license(lic)[0] == "CC-BY-SA"]
+    if total is None:
+        report.add(WARN, "Total row license", "no license cell in the Total row")
+    elif strongest and norm_license(total)[0] != "CC-BY-SA":
+        report.add(FAIL, "Total row license", f"{total} does not inherit the share-alike of {sorted(strongest)}")
+    else:
+        report.add(OK, "Total row license", f"{total} inherits share-alike from {len(strongest)} sources")
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__,
+                                     formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--tex", default=str(ROOT / "file.tex"), help="paper source")
+    parser.add_argument("--root", default=str(ROOT), help="repository root")
+    parser.add_argument("--ref", default=None,
+                        help="git ref for data/*.stats.json, or 'worktree' "
+                             "(default: the commit the .tex pins)")
+    args = parser.parse_args(argv)
+
+    root = Path(args.root).resolve()
+    tex = Path(args.tex).read_text(encoding="utf-8")
+    ref = args.ref or parse_inline_claims(tex)["pin"] or "HEAD"
+
+    report = Report()
+    check_trajectory(report, tex, root)
+    check_composition(report, tex, root, ref)
+    check_licenses(report, tex, root)
+    check_families(report, tex, root, ref)
+    return report.render()
+
+
+if __name__ == "__main__":
+    sys.exit(main())