Infini-News
Collection
A multilingual news corpus built from Common Crawl CC-News, indexed and queriable in milliseconds, cleaned an enriched with language and topic id • 2 items • Updated • 1
Dataset Preview
The full dataset viewer is not available (click to read why). Only showing a preview of the rows.
Commit 0/1 could not be created on the Hub (after 6 attempts).
Error code: CreateCommitError
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
release_id string | tool_commit string | exact_index_build dict | artifact_checks dict | api_checks dict | country dict | local_checks dict | limits list | annotation_packing dict | source_commit_map dict |
|---|---|---|---|---|---|---|---|---|---|
v2 | 0d989c5534ab8cf013b691b31b0008b73aa12b24 | {
"suffix_builder": "libsais64_omp",
"libsais_version": "2.10.4",
"libsais_commit": "ce90878d784b5ff7d019300535675e4a2e22aae0",
"libsais_license": "Apache-2.0",
"builder_license": "AGPL-3.0",
"public_source_commit": "0d989c5534ab8cf013b691b31b0008b73aa12b24",
"source_and_binary_sha256": {
"cc-news-ind... | {
"passed": true,
"release_id": "v2",
"document_count": 1372589195,
"shards": 117,
"article_files": 49748,
"annotations": {
"_lang": {
"files": 49748,
"bytes": 37666742715
},
"_iptc": {
"files": 49748,
"bytes": 33220526305
}
},
"index_bytes": 2886313629654,
"ret... | {
"passed": true,
"location": "isolated gateway and worker",
"release_id": "v2",
"document_count": 1372589195,
"shards": 117,
"metadata_samples": 117,
"filter_dimensions_checked": [
"languages",
"commonlingua_languages",
"topics",
"sources",
"countries"
],
"country_table_resolved":... | {
"rows": 103022,
"resolved": 58690,
"sha256": "71eca91f1d74d64fa8f364762cf8968d0be5408bebf37d525c2912cad8190c76",
"wikidata_dump": {
"path": "wikidata-20260526-all.json.gz",
"filename_date": "2026-05-26T00:00:00",
"size_bytes": 153730029490,
"mtime_ns": 1779825873000000000,
"sha256": "d1595... | {
"passed": true,
"python_tests_passed": 723,
"tested_source_commit": "0d989c5534ab8cf013b691b31b0008b73aa12b24",
"publisher_dependency_compatibility_tests_passed": 156,
"native_engine_tests_passed": 1,
"optional_R_DuckDB_UTC_fixture": "skipped: R DBI/duckdb unavailable"
} | [
"Sampled retrieval equality",
"Serving-host performance requires separate measurement",
"Human accuracy evaluation remains pending"
] | {
"language": {
"source_files": 49748,
"output_files": 117,
"rows": 1372589195,
"verification": "every output row group matched the source Arrow-value SHA-256",
"manifest_sha256": "cd565743a6e81754789244d1a55507f5cba25a6ebe2cefd2713b023a62b872e9"
},
"topic": {
"source_files": 49748,
"o... | {
"repository_url": "https://git.ksolovev.com/infini-news/",
"path": "release/source-commit-map.tsv",
"sha256": "92a8a1eee1032984e083c9d67e3041dc51d7351f7720eb6565a0d7250f4539f6",
"original_commit": "Commit recorded by the build or test provenance.",
"equivalent_commit": "Corresponding commit in the rewritten... |
Infini News FM Index v2
Release v2 indexes 1,372,589,195 articles from the matching corpus.
It contains 117 monthly shards, covering August 2016 through April 2026.
Files
ccnews_YYYY/shard_MM/{data.fm9,data_offset,meta.fm9,meta_offset}
metastore/ccnews_YYYY/shard_MM/{meta.zlib,meta_zlib_offset,.metadata-store.done}
release/{corpus-summary.json,shard_manifest.parquet,domains-geo.tsv}
engine/{engine-sa16.patch,engine-build.json,LICENSE}
The index and metadata store include their .release-manifests directories and completion markers.
The metadata store adds WARC identifiers, language predictions, and topic predictions to the index metadata.
Document offsets use the accepted corpus's sorted filenames and original row order.
Required engine
The text index uses suffix-array sampling density 16. The metadata index uses density 32.
Use the included density-16 engine build or apply engine/engine-sa16.patch to the recorded upstream commit before compiling.
The upstream density-32 text engine can open these files and return incorrect documents.
engine/engine-build.json records the source commit, source hashes, compiled extension hash, and tested runtime.
The supplied compiled extension targets CPython 3.12 on Linux x86_64. Other runtimes require a matching build.
The engine source retains its Apache-2.0 license. Dataset indexes and derived metadata use CC-BY 4.0.
The indexes use full suffix comparisons through libsais 2.10.4.
The exact suffix builder, dependency sources, licenses, and build identity are included under engine/.
The suffix builder uses AGPL-3.0 and libsais uses Apache-2.0.
Release identity and verification
Download the corpus and index by their recorded immutable Hugging Face commits.
Their corpus summaries and build manifests must identify release v2.
See release/verification.json for local validation and its sampling limits.
The metadata store was compared with the matching exact metadata indexes on 200 records per month.
Full article text and WARC identities were checked on three retrieved records per month.
The corpus summary omits local paths and records the original accepted-manifest digest. Build manifests retain that original digest.
The country table agrees with reference labels for 171 of 191 resolved hosts (89.5%) in a recovered 225-host set.
Eight hosts overlap the development set; independent human review remains undocumented.
The public query API moves to this release separately.
Check /api/v2/info for the loaded release. Production latency and throughput require measurements on the serving host.
The repository runbook describes worker startup, metadata validation, and release switching.
- Downloads last month
- 10,580