w3nabil/scholarly-metadata-corpus

Scholarly Metadata Corpus

A structured corpus of scholarly metadata collected from arXiv, designed for research on scholarly information retrieval, citation representation, bibliographic metadata, and LLM-based processing of academic literature.

Dataset Structure

The corpus is organized by source:

scholary-metadata-corpus/
├── arxiv/
│   ├── 2501.00001.json
│   ├── 2501.00002.json
│   ├── 2501.00003.json
│   ├── ...
│   └── 2501.xxxxx.json
└── ...

Each JSON file represents a single scholarly work identified by its arXiv identifier.

Note: This dataset is currently in a preliminary stage. The author retrieved the metadata using automated scripts, and the records have not yet undergone comprehensive manual purification or validation. The dataset may therefore contain extraction errors, inconsistencies, or incomplete metadata.

Record Format

Each arXiv record contains structured metadata associated with the corresponding HTML representation of the paper.

A typical record has the following structure:

{
  "arxiv_id": "2501.00001",
  "url": "https://arxiv.org/html/2501.00001",
  "from_cache": false,
  "status": "ok",
  "http_status": 200,
  "fetch_engine": "curl_cffi",
  "error": null,
  "title": "Mathematical modelling of flow and adsorption in a gas chromatograph",
  "authors": [
    "A. Cabrera-Codony",
    "A. Valverde",
    "K. Born",
    "O.A.I. Noreldin",
    "T.G. Myers"
  ],
  "authors_text": "...",
  "abstract": "..."
}

Fields

Field	Type	Description
arxiv_id	string	arXiv identifier of the paper
url	string	URL of the arXiv HTML representation
from_cache	boolean	Indicates whether the record was retrieved from a local/cache source
status	string	Retrieval status
http_status	integer	HTTP response status returned during retrieval
fetch_engine	string	Engine used to retrieve the document
error	string/null	Error information, if retrieval failed
title	string	Title of the scholarly work
authors	array	List of authors extracted from the document
authors_text	string	Author information as represented in the source
abstract	string	Abstract of the scholarly work

Records may contain additional fields as the corpus is extended.

Data Sources

The corpus currently uses the HTML representations provided by arXiv.

The arXiv HTML endpoint follows the general form:

https://arxiv.org/html/<arxiv_id>

The current release contains 1,100+ arXiv records and is planned to be expanded in future releases.

Planned Sources

Future versions of the corpus are planned to incorporate additional scholarly sources, including:

• arXiv
• DOI-indexed scholarly records
• Zenodo records

The addition of DOI and Zenodo records will allow the corpus to cover scholarly works beyond those available through arXiv and provide a broader basis for bibliographic and information-retrieval research.

Example

For an arXiv paper such as 2501.00001:

arxiv/
└── 2501.00001.json

The corresponding record can be loaded directly:

import json

with open("arxiv/2501.00001.json", "r", encoding="utf-8") as f:
    paper = json.load(f)

print(paper["title"])
print(paper["authors"])
print(paper["abstract"])

Limitations

The corpus represents metadata and extracted textual information available from the corresponding arXiv HTML representation. Consequently:

• Not every arXiv work necessarily has an available HTML representation.
• Extraction quality may vary between papers.
• Author formatting may differ between source documents.
• Retrieval metadata reflects the state of the source at the time of collection.
• A successful HTTP response does not necessarily imply that every element of the original paper was extracted perfectly.
• The current release has not undergone comprehensive manual validation or purification.

Intended Research Context

The corpus is particularly suited to experiments where scholarly metadata serves as a controlled information layer between academic documents and information-retrieval or language-model systems.

Because each record retains a persistent scholarly identifier, experimental outputs can be evaluated against deterministic identifiers such as arXiv IDs rather than relying exclusively on subjective LLM-based evaluation.

Potential applications include:

• Scholarly information retrieval
• Citation resolution
• Citation-format evaluation
• Bibliographic metadata extraction
• LLM-based retrieval evaluation
• Retrieval-augmented generation
• Academic corpus construction
• Citation-grounded question answering
• Tokenization and context-efficiency experiments

License and Content Ownership

The MIT license applies to the dataset's original organizational structure, metadata compilation, scripts, and other material created by the dataset author, where applicable.

The underlying scholarly works, papers, abstracts, and associated content are not owned by the author of this dataset. Such material remains subject to the copyrights, licenses, and terms applicable to the respective authors, publishers, repositories, and source platforms.

Users are responsible for complying with the applicable licenses and terms of use of the underlying sources and individual scholarly works.

If an author or rights holder believes that a record should be removed from the dataset, they may contact the dataset owner through the contact information provided in the repository.

The dataset author does not claim ownership of the underlying scholarly publications.

Dataset Status

This is an actively developing corpus. Future releases may include:

• Additional arXiv records
• DOI-based scholarly records
• Zenodo records
• Additional bibliographic metadata
• Citation representations
• Metadata normalization and purification
• Improved validation and deduplication
• Additional identifiers and persistent links

The structure and fields of the dataset may therefore change between releases.