Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders
Recent progress in text-to-image (T2I) diffusion models (DMs) has enabled high-quality visual synthesis from diverse textual prompts. Yet, most existing T2I DMs, even those equipped with large language model (LLM)-based text encoders, remain text-pixel mappers -- they employ LLMs merely as text encoders, without leveraging their inherent reasoning capabilities to infer what should be visually depicted given the textual prompt. To move beyond such literal generation, we propose the think-then-generate (T2G) paradigm, where the LLM-based text encoder is encouraged to reason about and rewrite raw user prompts; the states of the rewritten prompts then serve as diffusion conditioning. To achieve this, we first activate the think-then-rewrite pattern of the LLM encoder with a lightweight supervised fine-tuning process. Subsequently, the LLM encoder and diffusion backbone are co-optimized to ensure faithful reasoning about the context and accurate rendering of the semantics via Dual-GRPO. In particular, the text encoder is reinforced using image-grounded rewards to infer and recall world knowledge, while the diffusion backbone is pushed to produce semantically consistent and visually coherent images. Experiments show substantial improvements in factual consistency, semantic alignment, and visual realism across reasoning-based image generation and editing benchmarks, achieving 0.79 on WISE score, nearly on par with GPT-4. Our results constitute a promising step toward next-generation unified models with reasoning, expression, and demonstration capacities.
6969a7d132f0333869ff93bc
Siqi Kou
6969a7d132f0333869ff93bd
Jiachun Jin
claimed_verified
2026-04-03T09:09:45.939Z
66260b48d0a96a5db0de9722
/avatars/2901980e13b37af52cdb13bc5f8a6de3.svg
Jiachun Jin
orres
user
orres
6969a7d132f0333869ff93be
Zetong Zhou
6969a7d132f0333869ff93bf
Ye Ma
6969a7d132f0333869ff93c0
Yugang Wang
6969a7d132f0333869ff93c1
Quan Chen
6969a7d132f0333869ff93c2
Peng Jiang
6969a7d132f0333869ff93c3
Xiao Yang
6969a7d132f0333869ff93c4
Jun Zhu
6969a7d132f0333869ff93c5
Kai Yu
6969a7d132f0333869ff93c6
Zhijie Deng
Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoders
Recent progress in text-to-image (T2I) diffusion models (DMs) has enabled high-quality visual synthesis from diverse textual prompts. Yet, most existing T2I DMs, even those equipped with large language model (LLM)-based text encoders, remain text-pixel mappers -- they employ LLMs merely as text encoders, without leveraging their inherent reasoning capabilities to infer what should be visually depicted given the textual prompt. To move beyond such literal generation, we propose the think-then-generate (T2G) paradigm, where the LLM-based text encoder is encouraged to reason about and rewrite raw user prompts; the states of the rewritten prompts then serve as diffusion conditioning. To achieve this, we first activate the think-then-rewrite pattern of the LLM encoder with a lightweight supervised fine-tuning process. Subsequently, the LLM encoder and diffusion backbone are co-optimized to ensure faithful reasoning about the context and accurate rendering of the semantics via Dual-GRPO. In particular, the text encoder is reinforced using image-grounded rewards to infer and recall world knowledge, while the diffusion backbone is pushed to produce semantically consistent and visually coherent images. Experiments show substantial improvements in factual consistency, semantic alignment, and visual realism across reasoning-based image generation and editing benchmarks, achieving 0.79 on WISE score, nearly on par with GPT-4. Our results constitute a promising step toward next-generation unified models with reasoning, expression, and demonstration capacities.
Text-to-image diffusion models enhanced with language model reasoning capabilities achieve improved factual consistency and semantic alignment through a think-then-generate paradigm with dual-gradient reinforcement optimization.
text-to-image diffusion models language model-based text encoders think-then-generate paradigm supervised fine-tuning dual-gradient reinforcement optimization image-grounded rewards semantic alignment factual consistency visual realism