Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, a simple yet useful approach for better utilizing limited verifiable prompts targeting pass-rate-1 prompts. More specifically, Composition-RL automatically composes multiple problems into a new verifiable question and uses these compositional prompts for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. Codes, datasets, and models are available at https://github.com/XinXU-USTC/Composition-RL.
698eeb55cace060ff123af56
Xin Xu
claimed_verified
2026-02-13T09:36:02.476Z
64e2d169d2af12910d682130
https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg
xuxin
xx18
user
xx18
698eeb55cace060ff123af57
Clive Bai
698eeb55cace060ff123af58
Kai Yang
698eeb55cace060ff123af59
Tianhao Chen
admin_assigned
2026-03-06T15:47:13.001Z
6475a2da04c82116f9ba5744
/avatars/d596935f0e67eb959b60a4787d6738c4.svg
Sky
dandingsky
user
dandingsky
698eeb55cace060ff123af5a
Yangkun Chen
698eeb55cace060ff123af5b
Weijie Liu
698eeb55cace060ff123af5c
Hao Chen
698eeb55cace060ff123af5d
Yang Wang
698eeb55cace060ff123af5e
Saiyong Yang
698eeb55cace060ff123af5f
Can Yang
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, a simple yet useful approach for better utilizing limited verifiable prompts targeting pass-rate-1 prompts. More specifically, Composition-RL automatically composes multiple problems into a new verifiable question and uses these compositional prompts for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. Codes, datasets, and models are available at https://github.com/XinXU-USTC/Composition-RL.
Composition-RL improves reasoning capabilities by automatically composing multiple problems into new verifiable questions for reinforcement learning training.
Reinforcement Learning with Verifiable Rewards verifiable prompts pass rate compositional prompts curriculum learning cross-domain RL