SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning
Reinforcement learning (RL) offers a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve. In practice, RL progress often slows when task difficulty becomes poorly aligned with model capability, or when training is dominated by a narrow set of recurring problem patterns. To jointly address these issues, we propose SCALER (Synthetic sCalable Adaptive Learning Environment for Reasoning), a framework that sustains effective learning signals through adaptive environment design. SCALER introduces a scalable synthesis pipeline that converts real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation, enabling RL training beyond finite datasets while preserving strong correctness guarantees. Building on this, SCALER further employs an adaptive multi-environment RL strategy that dynamically adjusts instance difficulty and curates the active set of environments to track the model's capability frontier and maintain distributional diversity. This co-adaptation prevents reward sparsity, mitigates overfitting to narrow task patterns, and supports sustained improvement throughout training. Extensive experiments show that SCALER consistently outperforms dataset-based RL baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics.
696864120ac10a06522f6a43
Caijun Xu
admin_assigned
2026-01-15T19:08:52.927Z
677a4c4de0db74115dece89a
https://cdn-avatars.huggingface.co/v1/production/uploads/677a4c4de0db74115dece89a/MEUOgReiC6dW4TEH2aJAA.jpeg
Caijun Xu
SII-Molu
user
SII-Molu
696864120ac10a06522f6a44
Changyi Xiao
696864120ac10a06522f6a45
Zhongyuan Peng
696864120ac10a06522f6a46
Xinrun Wang
696864120ac10a06522f6a47
Yixin Cao
SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning
Reinforcement learning (RL) offers a principled way to enhance the reasoning capabilities of large language models, yet its effectiveness hinges on training signals that remain informative as models evolve. In practice, RL progress often slows when task difficulty becomes poorly aligned with model capability, or when training is dominated by a narrow set of recurring problem patterns. To jointly address these issues, we propose SCALER (Synthetic sCalable Adaptive Learning Environment for Reasoning), a framework that sustains effective learning signals through adaptive environment design. SCALER introduces a scalable synthesis pipeline that converts real-world programming problems into verifiable reasoning environments with controllable difficulty and unbounded instance generation, enabling RL training beyond finite datasets while preserving strong correctness guarantees. Building on this, SCALER further employs an adaptive multi-environment RL strategy that dynamically adjusts instance difficulty and curates the active set of environments to track the model's capability frontier and maintain distributional diversity. This co-adaptation prevents reward sparsity, mitigates overfitting to narrow task patterns, and supports sustained improvement throughout training. Extensive experiments show that SCALER consistently outperforms dataset-based RL baselines across diverse reasoning benchmarks and exhibits more stable, long-horizon training dynamics.
SCALER is a reinforcement learning framework that maintains effective training signals for language models through adaptive environment design and multi-environment strategies, enabling sustained performance improvements in reasoning tasks.
reinforcement learning large language models reasoning capabilities training signals synthetic environment adaptive environment design scalable synthesis pipeline verifiable reasoning environments controllable difficulty instance generation adaptive multi-environment RL capability frontier distributional diversity reward sparsity overfitting long-horizon training