DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels
Diffusion large language models (dLLMs) have emerged as a compelling alternative to autoregressive (AR) LLMs, owing to their capacity for parallel token generation. This paradigm is particularly well-suited for code generation, where holistic structural planning and non-sequential refinement are critical. Despite this potential, tailoring dLLMs for CUDA kernel generation remains challenging, obstructed not only by the high specialization but also by the severe lack of high-quality training data. To address these challenges, we construct CuKe, an augmented supervised fine-tuning dataset optimized for high-performance CUDA kernels. On top of it, we propose a bi-phase curated reinforcement learning (BiC-RL) framework consisting of a CUDA kernel infilling stage and an end-to-end CUDA kernel generation stage. Leveraging this training framework, we introduce DICE, a series of diffusion large language models designed for CUDA kernel generation, spanning three parameter scales, 1.7B, 4B, and 8B. Extensive experiments on KernelBench demonstrate that DICE significantly outperforms both autoregressive and diffusion LLMs of comparable scale, establishing a new state-of-the-art for CUDA kernel generation.
698e8f40cace060ff123ac49
Haolei Bai
claimed_verified
2026-02-13T09:36:44.960Z
66dbea44946bce6c94afac80
https://cdn-avatars.huggingface.co/v1/production/uploads/66dbea44946bce6c94afac80/MWL4AJEqs8XUVyEAX3QqN.png
Haolei Bai
DeadlyKitt3n
user
DeadlyKitt3n
698e8f40cace060ff123ac4a
Lingcheng Kong
698e8f40cace060ff123ac4b
Xueyi Chen
698e8f40cace060ff123ac4c
Jianmian Wang
698e8f40cace060ff123ac4d
Zhiqiang Tao
698e8f40cace060ff123ac4e
Huan Wang
DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels
Diffusion large language models (dLLMs) have emerged as a compelling alternative to autoregressive (AR) LLMs, owing to their capacity for parallel token generation. This paradigm is particularly well-suited for code generation, where holistic structural planning and non-sequential refinement are critical. Despite this potential, tailoring dLLMs for CUDA kernel generation remains challenging, obstructed not only by the high specialization but also by the severe lack of high-quality training data. To address these challenges, we construct CuKe, an augmented supervised fine-tuning dataset optimized for high-performance CUDA kernels. On top of it, we propose a bi-phase curated reinforcement learning (BiC-RL) framework consisting of a CUDA kernel infilling stage and an end-to-end CUDA kernel generation stage. Leveraging this training framework, we introduce DICE, a series of diffusion large language models designed for CUDA kernel generation, spanning three parameter scales, 1.7B, 4B, and 8B. Extensive experiments on KernelBench demonstrate that DICE significantly outperforms both autoregressive and diffusion LLMs of comparable scale, establishing a new state-of-the-art for CUDA kernel generation.
Diffusion large language models (dLLMs) for CUDA kernel generation achieve superior performance through a specialized dataset and reinforcement learning framework.
diffusion large language models autoregressive LLMs parallel token generation CUDA kernel generation supervised fine-tuning reinforcement learning bi-phase curated reinforcement learning kernel generation KernelBench