Yuhan123/cad-wave2-test-1689

CAD wave-2 test set (1,689-example sample)

A 1,689-example preference test set in the same 7-column schema as the internal cad_test.hf, built from wave 2 only of facebook/community-alignment-dataset (v1.1, revision 97343c7f63).

Sized deliberately to match cad_test.hf exactly (1,689 examples → 5,067 pairwise comparisons) so evaluation numbers are directly comparable at equal cost.

Schema

field	type	notes
prompt_id	int64	row identifier (non-sequential; carried over from the full 18,768-example set for traceability)
prompt	string	first-turn user message
responses	list[string]	always 4 candidate responses
labels	list[int64]	annotator votes, each an index 0–3 into responses — always length 1 here
maj_vote_label	int64	int(np.bincount(labels, minlength=4).argmax())
comparison_ids	list[int64]	source conversation_id
variance	float64	float(np.var(labels)) — always 0.0 here

Plus provenance columns ignored by the eval code: annotator_id, assigned_lang, annotator_country, turn_index.

How it was built

make_cad_wave2_test.py --sample 1689 --seed 2026

First turns only (wave 2 stores no conversation history, so turns 2–4 would be context-free). Deterministic: .shuffle(seed=2026).select(range(1689)).

Important caveats

1. Single-annotator labels. In wave 2, no two annotators ever judged the same four responses — every conversation got freshly generated candidates. So labels always has length 1, maj_vote_label is just that one vote, and variance is uniformly 0.0. Consequences:

• label_strategy="maj_vote" and "labels_0" are identical.
• label_strategy="labels_1" (or any labels_N, N>0) silently returns 0 for every example under the common example["labels"][N] if len(...) > N else 0 idiom — i.e. it asserts response A is preferred throughout. Do not use it with this dataset.
• No high/low-variance split is possible.

Pooled multi-annotator labels exist only in wave 1 of the source dataset (1,300 prompts with an identical shared response slate; 2,526 prompts with ≥10 annotators).

2. Noisier ground truth than majority-vote sets. Measured effect: a policy-likelihood pairwise eval scored 0.6250 on a ~8.9-annotator majority-vote set vs 0.4851 here — pooling ~9 annotators is worth roughly +14 points of apparent model accuracy. A lower score on this set does not mean a worse model.

3. Demographic skew. Wave 2 deliberately over-sampled groups under-represented in wave 1 (older annotators, lower formal education). Pooling it with wave-1 data shifts demographic marginals.

4. Prompt overlap. 294 of the 1,689 rows (17.4%) have a prompt that also appears in the wave-1 derived cad_test. The responses are disjoint (zero shared response text), so this is prompt-level rather than item-level overlap — but filter if you are training on one and evaluating on the other.

Composition

Languages: pt 404 · en 377 · fr 307 · hi 305 · it 296. maj_vote_label distribution: 0→503, 1→389, 2→398, 3→399. 434 distinct annotators, 1,634 distinct prompts.

Licence and attribution

CC-BY-4.0, inherited from the source dataset. Please cite:

@inproceedings{zhang2026cultivating,
  title     = {Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset},
  author    = {Lily H Zhang and Smitha Milli and Karen Long Jusko and Jonathan Smith and Brandon Amos and Wassim Bouaziz and Manon Revel and Jack Kussman and Yasha Sheynin and Lisa Titus and Bhaktipriya Radharapu and Jane Yu and Vidya Sarma and Kristopher Rose and Maximilian Nickel},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026},
  url       = {https://openreview.net/forum?id=4NtoAVqfhA}
}

Note: in the source dataset ~31% of conversations are annotator-initiated and unmoderated. Apply your own filtering before using this data for training.