Free Lunch for Pass@k? Low Cost Diverse Sampling for Diffusion Language Models
Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@k problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead, while ensuring that each sample contributes a unique perspective to the batch. We evaluate our method on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model. Our results demonstrate significantly improved diversity and Pass@k performance across various temperature settings. As a simple modification to the sampling process, our method offers an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search. We make our code available at https://github.com/sean-lamont/odd.
69af657bb3ab31dee1281b8f
Sean Lamont
claimed_verified
2026-03-10T09:45:24.371Z
659611e2580e8d6da12e742d
/avatars/4c2258894fb20e897ecce39e8f60d401.svg
Sean Lamont
sean-lamont
user
sean-lamont
69af657bb3ab31dee1281b90
Christian Walder
69af657bb3ab31dee1281b91
Paul Montague
69af657bb3ab31dee1281b92
Amir Dezfouli
69af657bb3ab31dee1281b93
Michael Norrish
Free Lunch for Pass@k? Low Cost Diverse Sampling for Diffusion Language Models
Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@k problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead, while ensuring that each sample contributes a unique perspective to the batch. We evaluate our method on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model. Our results demonstrate significantly improved diversity and Pass@k performance across various temperature settings. As a simple modification to the sampling process, our method offers an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search. We make our code available at https://github.com/sean-lamont/odd.
Diffusion language models suffer from redundant sampling, but a novel technique that repels samples from each other's feature space improves diversity and performance on code generation and math problem-solving tasks without additional computational cost.
diffusion language models autoregressive paradigm Pass@$k$ generative diversity sampling approaches redundant sampling feature space batch processing temperature settings HumanEval GSM8K LLaDA-8B-Instruct