HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified
69945fb23dd566a6e9cf9a24
Weiqi Zhai
admin_assigned
2026-02-18T16:12:42.765Z
68d29966b368904f7508f616
https://cdn-avatars.huggingface.co/v1/production/uploads/68d29966b368904f7508f616/SVSjwXynfvKxJEp2H6HV9.png
SKYLENAGE
skylenage
user
skylenage
69945fb23dd566a6e9cf9a25
Zhihai Wang
69945fb23dd566a6e9cf9a26
Jinghang Wang
69945fb23dd566a6e9cf9a27
Boyu Yang
69945fb23dd566a6e9cf9a28
Xiaogang Li
69945fb23dd566a6e9cf9a29
Xiang Xu
69945fb23dd566a6e9cf9a2a
Bohan Wang
69945fb23dd566a6e9cf9a2b
Peng Wang
69945fb23dd566a6e9cf9a2c
Xingzhe Wu
69945fb23dd566a6e9cf9a2d
Anfeng Li
69945fb23dd566a6e9cf9a2e
Qiyuan Feng
69945fb23dd566a6e9cf9a2f
Yuhao Zhou
69945fb23dd566a6e9cf9a30
Shoulin Han
69945fb23dd566a6e9cf9a31
Wenjie Luo
69945fb23dd566a6e9cf9a32
Yiyuan Li
69945fb23dd566a6e9cf9a33
Yaxuan Wang
69945fb23dd566a6e9cf9a34
Ruixian Luo
69945fb23dd566a6e9cf9a35
Guojie Lin
69945fb23dd566a6e9cf9a36
Peiyao Xiao
69945fb23dd566a6e9cf9a37
Chengliang Xu
69945fb23dd566a6e9cf9a38
Ben Wang
69945fb23dd566a6e9cf9a39
Zeyu Wang
69945fb23dd566a6e9cf9a3a
Zichao Chen
69945fb23dd566a6e9cf9a3b
Jianan Ye
69945fb23dd566a6e9cf9a3c
Yijie Hu
69945fb23dd566a6e9cf9a3d
Jialong Chen
69945fb23dd566a6e9cf9a3e
Zongwen Shen
69945fb23dd566a6e9cf9a3f
Yuliang Xu
69945fb23dd566a6e9cf9a40
An Yang
69945fb23dd566a6e9cf9a41
Bowen Yu
69945fb23dd566a6e9cf9a42
Dayiheng Liu
69945fb23dd566a6e9cf9a43
Junyang Lin
69945fb23dd566a6e9cf9a44
Hu Wei
69945fb23dd566a6e9cf9a45
Que Shen
69945fb23dd566a6e9cf9a46
Bing Zhao
HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified
HLE-Verified presents a validated and revised version of the HLE benchmark with improved reliability through expert review and model-based checks, demonstrating significant accuracy improvements in language model evaluations.
large language models benchmark evaluation verification protocol error taxonomy binary validation domain-expert review model-based cross-checks two-stage validation certified benchmark expert repairs model-assisted auditing final adjudication annotation noise model confidence