Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@k across large sampling budgets and increases the area under the pass@k curve (AUC@K) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.
6969b0a232f0333869ff946a
Zhiyuan Hu
admin_assigned
2026-01-16T15:32:38.232Z
64351475901c5734bcb64248
/avatars/12346d4301c1bfb00ce0ea128a93cc15.svg
Zhiyuan Hu
zhiyuanhucs
user
zhiyuanhucs
6969b0a232f0333869ff946b
Yucheng Wang
admin_assigned
2026-01-16T15:32:48.080Z
6891c906f3c31445cc040ab1
https://cdn-avatars.huggingface.co/v1/production/uploads/6891c906f3c31445cc040ab1/NBqxXOY7al4CD0XBj8ke2.jpeg
Yucheng Wang
DevilEnfant
user
DevilEnfant
6969b0a232f0333869ff946c
Yufei He
6969b0a232f0333869ff946d
Jiaying Wu
admin_assigned
2026-01-16T15:32:59.692Z
682deb444988bd82847e2b03
/avatars/15da087e84386ea72c6fa2db63571420.svg
Jia-Ying Wu
EricaWu
user
EricaWu
6969b0a232f0333869ff946e
Yilun Zhao
claimed_verified
2026-02-06T18:58:00.543Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
yilunzhao
user
yilunzhao
6969b0a232f0333869ff946f
See-Kiong Ng
6969b0a232f0333869ff9470
Cynthia Breazeal
admin_assigned
2026-01-16T15:33:06.327Z
672793ffa5255a517fd02045
/avatars/a2569be6f2e952b5b00e5d4b89a7cede.svg
Cynthia Breazeal
cynthiabreazeal
user
cynthiabreazeal
6969b0a232f0333869ff9471
Anh Tuan Luu
admin_assigned
2026-01-16T15:33:12.181Z
655722e80438e0854fae7554
/avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
Luu Anh Tuan
anhtuanluu36
user
anhtuanluu36
6969b0a232f0333869ff9472
Hae Won Park
admin_assigned
2026-01-16T15:33:17.979Z
682352cdb1c5350f850dd952
/avatars/5426efe0195ac8f914839e6585b1a112.svg
Hae Won Park
robohaewon
user
robohaewon
6969b0a232f0333869ff9473
Bryan Hooi
admin_assigned
2026-01-16T15:33:23.007Z
651d8032c50012d33e914f2f
/avatars/0a44c9f51fc50ce86582e328c361ea00.svg
Bryan Hooi
bhooi
user
bhooi
Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs
Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@k across large sampling budgets and increases the area under the pass@k curve (AUC@K) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale.
Reinforcement learning for large language models is enhanced by a rollout-level objective that rewards rare high-level reasoning strategies, improving diverse solution discovery without sacrificing initial performance.
reinforcement learning large language models exploration collapse pass@k pass@1 rollout-level objective high-level solution strategies clustering policy advantages AUC@K