Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models
Reinforcement learning enhances the reasoning capabilities of large language models but often involves high computational costs due to rollout-intensive optimization. Online prompt selection presents a plausible solution by prioritizing informative prompts to improve training efficiency. However, current methods either depend on costly, exact evaluations or construct prompt-specific predictive models lacking generalization across prompts. This study introduces Generalizable Predictive Prompt Selection (GPS), which performs Bayesian inference towards prompt difficulty using a lightweight generative model trained on the shared optimization history. Intermediate-difficulty prioritization and history-anchored diversity are incorporated into the batch acquisition principle to select informative prompt batches. The small predictive model also generalizes at test-time for efficient computational allocation. Experiments across varied reasoning benchmarks indicate GPS's substantial improvements in training efficiency, final performance, and test-time efficiency over superior baseline methods.
6981b8ebce18b186280964f1
Yun Qu
claimed_verified
2026-02-03T10:02:24.029Z
671f4c16692c424f3a28399b
/avatars/7750fee8b690331ddeaa755daa508771.svg
Yun Qu
yunqu
user
yunqu
6981b8ebce18b186280964f2
Qi Wang
6981b8ebce18b186280964f3
Yixiu Mao
6981b8ebce18b186280964f4
Heming Zou
6981b8ebce18b186280964f5
Yuhang Jiang
6981b8ebce18b186280964f6
Weijie Liu
6981b8ebce18b186280964f7
Clive Bai
6981b8ebce18b186280964f8
Kai Yang
6981b8ebce18b186280964f9
Yangkun Chen
6981b8ebce18b186280964fa
Saiyong Yang
6981b8ebce18b186280964fb
Xiangyang Ji
Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models
Reinforcement learning enhances the reasoning capabilities of large language models but often involves high computational costs due to rollout-intensive optimization. Online prompt selection presents a plausible solution by prioritizing informative prompts to improve training efficiency. However, current methods either depend on costly, exact evaluations or construct prompt-specific predictive models lacking generalization across prompts. This study introduces Generalizable Predictive Prompt Selection (GPS), which performs Bayesian inference towards prompt difficulty using a lightweight generative model trained on the shared optimization history. Intermediate-difficulty prioritization and history-anchored diversity are incorporated into the batch acquisition principle to select informative prompt batches. The small predictive model also generalizes at test-time for efficient computational allocation. Experiments across varied reasoning benchmarks indicate GPS's substantial improvements in training efficiency, final performance, and test-time efficiency over superior baseline methods.
Generalizable Predictive Prompt Selection (GPS) uses Bayesian inference with a lightweight generative model to efficiently select informative prompts for reinforcement learning-enhanced language models, improving training efficiency and performance.
reinforcement learning large language models prompt selection Bayesian inference generative model rollout-intensive optimization prompt difficulty batch acquisition test-time efficiency training efficiency