FlashSampling: Fast and Memory-Efficient Exact Sampling
Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We present FlashSampling, an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in HBM. The method is simple: compute logits tile-by-tile on chip, add Gumbel noise, keep only one maximizer per row and per vocabulary tile, and finish with a small reduction over tiles. The fused tiled kernel is exact because argmax decomposes over a partition; grouped variants for online and tensor-parallel settings are exact by hierarchical factorization of the categorical distribution. Across H100, H200, B200, and B300 GPUs, FlashSampling speeds up kernel-level decode workloads, and in end-to-end vLLM experiments, it reduces time per output token by up to 19% on the models we test. These results show that exact sampling, with no approximation, can be integrated into the matmul itself, turning a bandwidth-bound postprocessing step into a lightweight epilogue. Project Page: https://github.com/FlashSampling/FlashSampling.
69ba06245d2b9c28ea09f026
Tomas Ruiz
claimed_verified
2026-03-23T10:58:56.424Z
66588adb2889d3c69c24ec38
https://cdn-avatars.huggingface.co/v1/production/uploads/66588adb2889d3c69c24ec38/4C-5ZfKN6aTiRTYtiP2s3.jpeg
Tomas Ruiz
tomasruiz
user
tomasruiz
69ba06245d2b9c28ea09f027
Zhen Qin
69ba06245d2b9c28ea09f028
Yifan Zhang
claimed_verified
2026-03-18T08:45:18.928Z
647bf082aba7062fe5c51ca9
https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg
Yifan Zhang
yifAI
user
yifAI
69ba06245d2b9c28ea09f029
Xuyang Shen
69ba06245d2b9c28ea09f02a
Yiran Zhong
69ba06245d2b9c28ea09f02b
Mengdi Wang
FlashSampling: Fast and Memory-Efficient Exact Sampling
Sampling from a categorical distribution is mathematically simple, but in large-vocabulary decoding, it often triggers extra memory traffic and extra kernels after the LM head. We present FlashSampling, an exact sampling primitive that fuses sampling into the LM-head matmul and never materializes the logits tensor in HBM. The method is simple: compute logits tile-by-tile on chip, add Gumbel noise, keep only one maximizer per row and per vocabulary tile, and finish with a small reduction over tiles. The fused tiled kernel is exact because argmax decomposes over a partition; grouped variants for online and tensor-parallel settings are exact by hierarchical factorization of the categorical distribution. Across H100, H200, B200, and B300 GPUs, FlashSampling speeds up kernel-level decode workloads, and in end-to-end vLLM experiments, it reduces time per output token by up to 19% on the models we test. These results show that exact sampling, with no approximation, can be integrated into the matmul itself, turning a bandwidth-bound postprocessing step into a lightweight epilogue. Project Page: https://github.com/FlashSampling/FlashSampling.
FlashSampling enables efficient categorical sampling by fusing the operation into the language model head matmul, eliminating memory overhead and reducing decoding time by up to 19%.
categorical distribution Gumbel noise argmax tiled kernel language model head matmul bandwidth-bound epilogue