Reasoning with Sampling: Your Base Model is Smarter Than You Think
Frontier reasoning models have exhibited incredible capabilities across a
wide array of disciplines, driven by posttraining large language models (LLMs)
with reinforcement learning (RL). However, despite the widespread success of
this paradigm, much of the literature has been devoted to disentangling truly
novel behaviors that emerge during RL but are not present in the base models.
In our work, we approach this question from a different angle, instead asking
whether comparable reasoning capabilites can be elicited from base models at
inference time by pure sampling, without any additional training. Inspired by
Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened
distributions, we propose a simple iterative sampling algorithm leveraging the
base models' own likelihoods. Over different base models, we show that our
algorithm offers substantial boosts in reasoning that nearly match and even
outperform those from RL on a wide variety of single-shot tasks, including
MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in
diversity over multiple samples that is characteristic of RL-posttraining.
Crucially, our method does not require training, curated datasets, or a
verifier, suggesting broad applicability beyond easily verifiable domains.
68fa7a16f158a71c5a2f56f9
Aayush Karan
68fa7a16f158a71c5a2f56fa
Yilun Du
Reasoning with Sampling: Your Base Model is Smarter Than You Think
Frontier reasoning models have exhibited incredible capabilities across a
wide array of disciplines, driven by posttraining large language models (LLMs)
with reinforcement learning (RL). However, despite the widespread success of
this paradigm, much of the literature has been devoted to disentangling truly
novel behaviors that emerge during RL but are not present in the base models.
In our work, we approach this question from a different angle, instead asking
whether comparable reasoning capabilites can be elicited from base models at
inference time by pure sampling, without any additional training. Inspired by
Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened
distributions, we propose a simple iterative sampling algorithm leveraging the
base models' own likelihoods. Over different base models, we show that our
algorithm offers substantial boosts in reasoning that nearly match and even
outperform those from RL on a wide variety of single-shot tasks, including
MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in
diversity over multiple samples that is characteristic of RL-posttraining.
Crucially, our method does not require training, curated datasets, or a
verifier, suggesting broad applicability beyond easily verifiable domains.
An iterative sampling algorithm enhances reasoning capabilities in base models without additional training, matching or outperforming reinforcement learning on single-shot tasks.
posttraining large language models reinforcement learning Markov chain Monte Carlo iterative sampling algorithm likelihoods MATH500 HumanEval GPQA diversity collapse