Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
Accurate estimation of item (question or task) difficulty is critical for educational assessment but suffers from the cold start problem. While Large Language Models demonstrate superhuman problem-solving capabilities, it remains an open question whether they can perceive the cognitive struggles of human learners. In this work, we present a large-scale empirical analysis of Human-AI Difficulty Alignment for over 20 models across diverse domains such as medical knowledge and mathematical reasoning. Our findings reveal a systematic misalignment where scaling up model size is not reliably helpful; instead of aligning with humans, models converge toward a shared machine consensus. We observe that high performance often impedes accurate difficulty estimation, as models struggle to simulate the capability limitations of students even when being explicitly prompted to adopt specific proficiency levels. Furthermore, we identify a critical lack of introspection, as models fail to predict their own limitations. These results suggest that general problem-solving capability does not imply an understanding of human cognitive struggles, highlighting the challenge of using current models for automated difficulty prediction.
694a0eb6335742716e932275
Ming Li
694a0eb6335742716e932276
Han Chen
694a0eb6335742716e932277
Yunze Xiao
claimed_verified
2026-01-14T12:47:52.599Z
67c8af4885b7a86d7515b77c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/4bRu0MKg6voGX5fyqninn.png
Lorenzo Xiao
lrzneedresearch
user
lrzneedresearch
694a0eb6335742716e932278
Jian Chen
claimed_verified
2025-12-25T20:48:39.363Z
64593836c16ecb4815e082ce
https://cdn-avatars.huggingface.co/v1/production/uploads/64593836c16ecb4815e082ce/cHv6bOowt4N7n-dweNozO.jpeg
Jian Chen
puar-playground
user
puar-playground
694a0eb6335742716e932279
Hong Jiao
claimed_verified
2025-12-25T20:48:36.913Z
694a13577a9d7051d14f0f44
/avatars/2654c1af436507870c4f1bb64aa0823b.svg
Hong Jiao
hjiao
user
hjiao
694a0eb6335742716e93227a
Tianyi Zhou
claimed_verified
2025-12-25T20:48:41.857Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
zhoutianyi
user
zhoutianyi
Can LLMs Estimate Student Struggles? Human-AI Difficulty Alignment with Proficiency Simulation for Item Difficulty Prediction
Accurate estimation of item (question or task) difficulty is critical for educational assessment but suffers from the cold start problem. While Large Language Models demonstrate superhuman problem-solving capabilities, it remains an open question whether they can perceive the cognitive struggles of human learners. In this work, we present a large-scale empirical analysis of Human-AI Difficulty Alignment for over 20 models across diverse domains such as medical knowledge and mathematical reasoning. Our findings reveal a systematic misalignment where scaling up model size is not reliably helpful; instead of aligning with humans, models converge toward a shared machine consensus. We observe that high performance often impedes accurate difficulty estimation, as models struggle to simulate the capability limitations of students even when being explicitly prompted to adopt specific proficiency levels. Furthermore, we identify a critical lack of introspection, as models fail to predict their own limitations. These results suggest that general problem-solving capability does not imply an understanding of human cognitive struggles, highlighting the challenge of using current models for automated difficulty prediction.
Large Language Models struggle to accurately estimate human cognitive difficulty due to a misalignment with human perceptions and a lack of introspection regarding their own limitations.
Human-AI Difficulty Alignment Large Language Models cognitive struggles machine consensus proficiency levels introspection