CoffeeGitta/difficulty-gsm8k-generations

Generations Dataset: gsm8k

LLM-generated solutions across train/validation/test splits for multiple models.

Columns

Column	Type	Description
problem	str	Problem statement
generated_solutions	list	Generated solutions with scores
success_rate	float	Fraction of correct generations
majority_vote_is_correct	int (0/1)	Whether majority vote is correct
k	int	Number of samples generated
temperature	float	Sampling temperature
max_len	int	Maximum generation length
model_name	str	Model used for generation

Usage

from datasets import load_dataset

dataset = load_dataset("CoffeeGitta/difficulty-gsm8k-generations", name="<org--model>")
train = dataset["train"]

Citation

@article{lugoloobi_llms_2026,
    title = {LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations},
    url = {http://arxiv.org/abs/2602.09924},
    author = {Lugoloobi, William and Foster, Thomas and Bankes, William and Russell, Chris},
    year = {2026},
}