ngqtrung/hopchain-v5

HopChain v5

22,550 multi-hop video reasoning questions over 13,378 videos, plus a curated 1,000-question validation split, for RL with verifiable rewards. Each question forces the model to look at several separated moments in one video, read one fact at each, and sum them. Every answer is a single integer, so grading is exact and needs no judge model.

Contents

split	rows	videos
train	22,550	13,378
validation	1,000	1,000 (one question each)

Sources: longvila 10,488 · finevideo 8,047 · vript 4,015 (train). Hops per question: 4 hops 12,604 · 5 hops 7,527 · 6 hops 1,824 · 3 hops 595. Hop types: order · spatial · action · color. Gold answers are integers, median 179.

The validation split

One question per video, stratified so no single property dominates:

axis	distribution
hops 4 / 5 / 6	302 / 450 / 248
video length (caption segments) <40 / 40-70 / 70-110 / >110	151 / 301 / 300 / 248
source	longvila 467 · finevideo 358 · vript 175
distinct hop types per question	2: 213 · 3: 554 · 4: 233

Eligibility was >= 4 hops and >= 2 distinct hop types, which removes the shortest and most repetitive questions.

This split is not difficulty-calibrated. solve_rate is -1.0 (unknown) on every row — no solve-rate probe has been run. The selection above is a structural proxy for difficulty, not a measurement against a model.

Design

• Sum only. The combine is A+B+C+…, never multiplication, so difficulty lives in perception rather than arithmetic. At generation every one of the 2^n branch combinations is checked to give a distinct total, so a right answer cannot come from two errors cancelling, and a wrong total often identifies which hop was missed.
• Exact grading. Every row carries extra_info.tolerance = 0.0. A 1% relative tolerance would accept a neighbouring branch total on 42.6% of these questions.
• Grounded. Questions are written from dense caption timelines, and each hop is checked against the caption by a separate judge pass. Unsupported hops are dropped, never repaired by a model.

Video decode contract

Rows carry fps=1, max_frames=140, max_pixels=50176, min_pixels=3136, and the video field points at a local path — supply your own decoded videos. Prompts are two messages: a fixed system prompt and a user turn beginning with a <video> placeholder.

Known limitations

• The grounding judge reads a fixed ~18-segment window, so on long captions it rejects some sound questions for evidence it never saw: rewrite rate is 8.9% where the window covers the whole caption and 39.5% at 46-70 segments. This costs recall, not precision — about 12% of candidates were dropped without a fault.
• A hand review of 20 questions found 18 clean; both defects were foreground/background hops whose depth claim the caption never states.
• order hops appear in ~99% of questions. The type mix is uneven by construction.