Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers
Recent DiT-based text-to-image models increasingly adopt LLMs as text encoders, yet text conditioning remains largely static and often utilizes only a single LLM layer, despite pronounced semantic hierarchy across LLM layers and non-stationary denoising dynamics over both diffusion time and network depth. To better match the dynamic process of DiT generation and thereby enhance the diffusion model's generative capability, we introduce a unified normalized convex fusion framework equipped with lightweight gates to systematically organize multi-layer LLM hidden states via time-wise, depth-wise, and joint fusion. Experiments establish Depth-wise Semantic Routing as the superior conditioning strategy, consistently improving text-image alignment and compositional generation (e.g., +9.97 on the GenAI-Bench Counting task). Conversely, we find that purely time-wise fusion can paradoxically degrade visual generation fidelity. We attribute this to a train-inference trajectory mismatch: under classifier-free guidance, nominal timesteps fail to track the effective SNR, causing semantically mistimed feature injection during inference. Overall, our results position depth-wise routing as a strong and effective baseline and highlight the critical need for trajectory-aware signals to enable robust time-dependent conditioning.
69843228e34659da7e1f5060
Bozhou Li
claimed_verified
2026-04-08T08:07:14.804Z
661e62c6bac5d981f886f77b
/avatars/f1eb51ed4499ca434c8939573dfbd5e2.svg
Bozhou Li
zooblastlbz
user
zooblastlbz
69843228e34659da7e1f5061
Yushuo Guan
claimed_verified
2026-02-05T10:52:24.319Z
66ac46766c3f950f4f10b9f9
/avatars/027b573bc6e5b18107e762645cec6069.svg
Yushuo Guan
UnnamedWatcher
user
UnnamedWatcher
69843228e34659da7e1f5062
Haolin Li
claimed_verified
2026-02-05T10:52:21.747Z
64858cbb6121946cf1e3d41b
/avatars/cbe80473069512ba74e9713f8fa0942b.svg
lihaolin
tdlhl
user
tdlhl
69843228e34659da7e1f5063
Bohan Zeng
claimed_verified
2026-04-07T08:49:15.304Z
6671214c92412fd4640714eb
/avatars/48fa84e7bc3bb92ad0192aa26b32de10.svg
bohan zeng
zbhpku
user
zbhpku
69843228e34659da7e1f5064
Yiyan Ji
69843228e34659da7e1f5065
Yue Ding
69843228e34659da7e1f5066
Pengfei Wan
69843228e34659da7e1f5067
Kun Gai
69843228e34659da7e1f5068
Yuanxing Zhang
69843228e34659da7e1f5069
Wentao Zhang
Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers
Recent DiT-based text-to-image models increasingly adopt LLMs as text encoders, yet text conditioning remains largely static and often utilizes only a single LLM layer, despite pronounced semantic hierarchy across LLM layers and non-stationary denoising dynamics over both diffusion time and network depth. To better match the dynamic process of DiT generation and thereby enhance the diffusion model's generative capability, we introduce a unified normalized convex fusion framework equipped with lightweight gates to systematically organize multi-layer LLM hidden states via time-wise, depth-wise, and joint fusion. Experiments establish Depth-wise Semantic Routing as the superior conditioning strategy, consistently improving text-image alignment and compositional generation (e.g., +9.97 on the GenAI-Bench Counting task). Conversely, we find that purely time-wise fusion can paradoxically degrade visual generation fidelity. We attribute this to a train-inference trajectory mismatch: under classifier-free guidance, nominal timesteps fail to track the effective SNR, causing semantically mistimed feature injection during inference. Overall, our results position depth-wise routing as a strong and effective baseline and highlight the critical need for trajectory-aware signals to enable robust time-dependent conditioning.
Text conditioning in DiT-based models is enhanced through a unified normalized convex fusion framework that optimizes multi-layer LLM hidden states via depth-wise semantic routing, improving text-image alignment and compositional generation.
DiT-based text-to-image models LLMs text encoders diffusion models normalized convex fusion lightweight gates multi-layer LLM hidden states time-wise fusion depth-wise fusion joint fusion classifier-free guidance effective SNR semantic hierarchy generative capability text-image alignment compositional generation Depth-wise Semantic Routing train-inference trajectory mismatch