C^2DLM: Causal Concept-Guided Diffusion Large Language Models
Autoregressive (AR) language models and Diffusion Language Models (DLMs) constitute the two principal paradigms of large language models. However, both paradigms suffer from insufficient reasoning capabilities. Human reasoning inherently relies on causal knowledge and thought, which are reflected in natural language. But in the AR paradigm, language is modeled as next token prediction (a strictly left-to-right, token-by-token order), whereas natural language itself exhibits more flexible causal structures. In the DLM paradigm, the attention mechanism is fully connected, which entirely disregards causal order. To fill this gap, we propose a \textbf{C}ausal \textbf{C}oncept-Guided \textbf{D}iffusion \textbf{L}anguage \textbf{M}odel (C^2DLM). Starting from DLM's fully connected attention, C^2DLM first obtains a concept-level causal graph from the teacher model, and then explicitly guides attention to learn causal relationships between concepts. By focusing on causal relationships and avoiding interference from difficult subgoals involving causal inversion, C^2DLM improves 12\% with about 3.2 times training speedup in the COT-OrderPerturb task, and achieves an average gain of 1.31\% across six downstream reasoning tasks. More details in the repository ~https://github.com/Kairong-Han/C-2-DLM{here}.
692fd6c726742347f61dad69
Kairong Han
claimed_verified
2025-12-04T08:49:07.536Z
68c6b73ad0bef8d72febf45d
/avatars/7b65febb59f2524a81b08c9f6bd881f5.svg
Kairong Han
Kairong-Han
user
Kairong-Han
692fd6c726742347f61dad6a
Nuanqiao Shan
692fd6c726742347f61dad6b
Ziyu Zhao
692fd6c726742347f61dad6c
Zijing Hu
692fd6c726742347f61dad6d
Xinpeng Dong
692fd6c726742347f61dad6e
Junjian Ye
692fd6c726742347f61dad6f
Lujia Pan
692fd6c726742347f61dad70
Fei Wu
692fd6c726742347f61dad71
Kun Kuang
C^2DLM: Causal Concept-Guided Diffusion Large Language Models
Autoregressive (AR) language models and Diffusion Language Models (DLMs) constitute the two principal paradigms of large language models. However, both paradigms suffer from insufficient reasoning capabilities. Human reasoning inherently relies on causal knowledge and thought, which are reflected in natural language. But in the AR paradigm, language is modeled as next token prediction (a strictly left-to-right, token-by-token order), whereas natural language itself exhibits more flexible causal structures. In the DLM paradigm, the attention mechanism is fully connected, which entirely disregards causal order. To fill this gap, we propose a \textbf{C}ausal \textbf{C}oncept-Guided \textbf{D}iffusion \textbf{L}anguage \textbf{M}odel (C^2DLM). Starting from DLM's fully connected attention, C^2DLM first obtains a concept-level causal graph from the teacher model, and then explicitly guides attention to learn causal relationships between concepts. By focusing on causal relationships and avoiding interference from difficult subgoals involving causal inversion, C^2DLM improves 12\% with about 3.2 times training speedup in the COT-OrderPerturb task, and achieves an average gain of 1.31\% across six downstream reasoning tasks. More details in the repository ~https://github.com/Kairong-Han/C-2-DLM{here}.
A Causal Concept-Guided Diffusion Language Model (C2DLM) improves reasoning capabilities by learning causal relationships between concepts, enhancing performance and training efficiency in downstream tasks.
autoregressive models diffusion language models causal knowledge next token prediction fully connected attention causal graph causal relationships causal inversion COT-OrderPerturb task downstream reasoning tasks