The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N
  Sampling via max@k Optimisation
The application of Reinforcement Learning with Verifiable Rewards (RLVR) to
mathematical and coding domains has demonstrated significant improvements in
the reasoning and problem-solving abilities of Large Language Models. Despite
its success in single generation problem solving, the reinforcement learning
fine-tuning process may harm the model's exploration ability, as reflected in
decreased diversity of generations and a resulting degradation of performance
during Best-of-N sampling for large N values. In this work, we focus on
optimizing the max@k metric, a continuous generalization of pass@k. We derive
an unbiased on-policy gradient estimate for direct optimization of this metric.
Furthermore, we extend our derivations to the off-policy updates, a common
element in modern RLVR algorithms, that allows better sample efficiency.
Empirically, we show that our objective effectively optimizes max@k metric in
off-policy scenarios, aligning the model with the Best-of-N inference strategy.
69008b6a646208eac0d1ef43
Farid Bagirov
claimed_verified
2025-10-28T15:19:43.421Z
62ecd8a0b72a69615d6bbd76
/avatars/73c47466cbd3ae217b23e5e76bd0f740.svg
Farid Bagirov
kraalfar
user
kraalfar
69008b6a646208eac0d1ef44
Mikhail Arkhipov
69008b6a646208eac0d1ef45
Ksenia Sycheva
69008b6a646208eac0d1ef46
Evgeniy Glukhov
claimed_verified
2025-10-28T14:32:12.590Z
6440ff39ad24e9b2cfba8575
https://cdn-avatars.huggingface.co/v1/production/uploads/6440ff39ad24e9b2cfba8575/QoQC60vbfd2YzTtPAXpM-.jpeg
Evgeniy Glukhov
jenyag
user
jenyag
69008b6a646208eac0d1ef47
Egor Bogomolov
claimed_verified
2025-10-28T15:19:40.431Z
64380bed961bb61e463bf93d
https://cdn-avatars.huggingface.co/v1/production/uploads/64380bed961bb61e463bf93d/zsel0Dzv1yU9O8zAxvCBw.jpeg
Egor Bogomolov
egor-bogomolov
user
egor-bogomolov
The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N
  Sampling via max@k Optimisation
The application of Reinforcement Learning with Verifiable Rewards (RLVR) to
mathematical and coding domains has demonstrated significant improvements in
the reasoning and problem-solving abilities of Large Language Models. Despite
its success in single generation problem solving, the reinforcement learning
fine-tuning process may harm the model's exploration ability, as reflected in
decreased diversity of generations and a resulting degradation of performance
during Best-of-N sampling for large N values. In this work, we focus on
optimizing the max@k metric, a continuous generalization of pass@k. We derive
an unbiased on-policy gradient estimate for direct optimization of this metric.
Furthermore, we extend our derivations to the off-policy updates, a common
element in modern RLVR algorithms, that allows better sample efficiency.
Empirically, we show that our objective effectively optimizes max@k metric in
off-policy scenarios, aligning the model with the Best-of-N inference strategy.
Optimizing the max@k metric through unbiased gradient estimates in RLVR improves the diversity and performance of Large Language Models in Best-of-N sampling scenarios.
Reinforcement Learning with Verifiable Rewards (RLVR) Large Language Models max@k metric pass@k on-policy gradient estimate off-policy updates Best-of-N sampling