Revisiting Generalization Across Difficulty Levels: It's Not So Easy
We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.
6927cf95243b2216fb75cd90
Yeganeh Kordi
claimed_verified
2025-11-27T09:58:38.604Z
626d13b15f7327906f0514e9
/avatars/6fb9b22961b9a283700c60713bc5a61f.svg
Yeganeh Kordi
Yeganeh
user
Yeganeh
6927cf95243b2216fb75cd91
Nihal V. Nayak
claimed_verified
2025-12-01T16:24:50.278Z
60d0e7ff0c9ba111563b81d7
https://cdn-avatars.huggingface.co/v1/production/uploads/60d0e7ff0c9ba111563b81d7/MudCH3WtpBIQGuodc7GND.jpeg
Nihal Nayak
nihalnayak
user
nihalnayak
6927cf95243b2216fb75cd92
Max Zuo
6927cf95243b2216fb75cd93
Ilana Nguyen
6927cf95243b2216fb75cd94
Stephen H. Bach
Revisiting Generalization Across Difficulty Levels: It's Not So Easy
We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.
LLMs do not consistently generalize across different task difficulties, indicating the need for a broad range of difficulty levels in both training and evaluation datasets.
large language models LLMs generalization task difficulties data curation evaluation Item Response Theory IRT cross-difficulty generalization