CL-bench: A Benchmark for Context Learning
Current language models (LMs) excel at reasoning over prompts using pre-trained knowledge. However, real-world tasks are far more complex and context-dependent: models must learn from task-specific context and leverage new knowledge beyond what is learned during pre-training to reason and resolve tasks. We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked. To this end, we introduce CL-bench, a real-world benchmark consisting of 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. Each task is designed such that the new content required to resolve it is contained within the corresponding context. Resolving tasks in CL-bench requires models to learn from the context, ranging from new domain-specific knowledge, rule systems, and complex procedures to laws derived from empirical data, all of which are absent from pre-training. This goes far beyond long-context tasks that primarily test retrieval or reading comprehension, and in-context learning tasks, where models learn simple task patterns via instructions and demonstrations. Our evaluations of ten frontier LMs find that models solve only 17.2% of tasks on average. Even the best-performing model, GPT-5.1, solves only 23.7%, revealing that LMs have yet to achieve effective context learning, which poses a critical bottleneck for tackling real-world, complex context-dependent tasks. CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
6982e28a9084cb4f0ecb5847
Shihan Dou
6982e28a9084cb4f0ecb5848
Ming Zhang
claimed_verified
2026-02-04T19:03:53.545Z
65b71c0582d38451342f7334
/avatars/f9763a0ac361c350e6c6732e23564567.svg
Ming Zhang
konglongge
user
konglongge
6982e28a9084cb4f0ecb5849
Zhangyue Yin
6982e28a9084cb4f0ecb584a
Chenhao Huang
6982e28a9084cb4f0ecb584b
Yujiong Shen
6982e28a9084cb4f0ecb584c
Junzhe Wang
6982e28a9084cb4f0ecb584d
Jiayi Chen
6982e28a9084cb4f0ecb584e
Yuchen Ni
6982e28a9084cb4f0ecb584f
Junjie Ye
claimed_verified
2026-02-27T16:45:54.735Z
66384be673c2c55f2ded89fa
/avatars/1d8721074f0f51fab405f81474f2035f.svg
Junjie Ye
Junjie-Ye
user
Junjie-Ye
6982e28a9084cb4f0ecb5850
Cheng Zhang
6982e28a9084cb4f0ecb5851
Huaibing Xie
6982e28a9084cb4f0ecb5852
Jianglu Hu
6982e28a9084cb4f0ecb5853
Shaolei Wang
6982e28a9084cb4f0ecb5854
Weichao Wang
6982e28a9084cb4f0ecb5855
Yanling Xiao
6982e28a9084cb4f0ecb5856
Yiting Liu
6982e28a9084cb4f0ecb5857
Zenan Xu
6982e28a9084cb4f0ecb5858
Zhen Guo
6982e28a9084cb4f0ecb5859
Pluto Zhou
6982e28a9084cb4f0ecb585a
Tao Gui
6982e28a9084cb4f0ecb585b
Zuxuan Wu
6982e28a9084cb4f0ecb585c
Xipeng Qiu
6982e28a9084cb4f0ecb585d
Qi Zhang
6982e28a9084cb4f0ecb585e
Xuanjing Huang
6982e28a9084cb4f0ecb585f
Yu-Gang Jiang
6982e28a9084cb4f0ecb5860
Di Wang
6982e28a9084cb4f0ecb5861
Shunyu Yao
CL-bench: A Benchmark for Context Learning
Current language models (LMs) excel at reasoning over prompts using pre-trained knowledge. However, real-world tasks are far more complex and context-dependent: models must learn from task-specific context and leverage new knowledge beyond what is learned during pre-training to reason and resolve tasks. We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked. To this end, we introduce CL-bench, a real-world benchmark consisting of 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. Each task is designed such that the new content required to resolve it is contained within the corresponding context. Resolving tasks in CL-bench requires models to learn from the context, ranging from new domain-specific knowledge, rule systems, and complex procedures to laws derived from empirical data, all of which are absent from pre-training. This goes far beyond long-context tasks that primarily test retrieval or reading comprehension, and in-context learning tasks, where models learn simple task patterns via instructions and demonstrations. Our evaluations of ten frontier LMs find that models solve only 17.2% of tasks on average. Even the best-performing model, GPT-5.1, solves only 23.7%, revealing that LMs have yet to achieve effective context learning, which poses a critical bottleneck for tackling real-world, complex context-dependent tasks. CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
Language models struggle with context learning, requiring new knowledge and reasoning beyond pre-training, as demonstrated by a comprehensive benchmark revealing poor performance on real-world tasks.
language models context learning pre-trained knowledge real-world tasks in-context learning long-context tasks retrieval reading comprehension task patterns frontier LMs GPT-5.1