Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards
Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.
698da10c65c0d15a6d162353
Kirill Pavlenko
698da10c65c0d15a6d162354
Alexander Golubev
claimed_verified
2026-02-12T11:55:59.641Z
644e9ffcd6001776ed77d874
/avatars/b93e02caf929679b7e9bc589eed0b689.svg
Alexander
djalexj
user
djalexj
698da10c65c0d15a6d162355
Simon Karasik
698da10c65c0d15a6d162356
Boris Yangel
Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards
Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.
Blockwise Advantage Estimation addresses reward interference in structured generations by assigning separate advantages to different text blocks, using outcome-conditioned baselines to avoid expensive nested rollouts.
Group Relative Policy Optimization advantage estimation reward interference structured generations text blocks outcome-conditioned baseline nested rollouts confidence-weighted ensembling