Rethinking Global Text Conditioning in Diffusion Transformers
Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance, suggesting that attention alone is generally sufficient for faithfully propagating prompt information. However, we reveal that the pooled embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various diffusion models, bringing improvements across diverse tasks, including text-to-image/video generation and image editing.
698c48066052d3bed9630d0f
Nikita Starodubcev
698c48066052d3bed9630d10
Daniil Pakhomov
698c48066052d3bed9630d11
Zongze Wu
698c48066052d3bed9630d12
Ilya Drobyshevskiy
698c48066052d3bed9630d13
Yuchen Liu
698c48066052d3bed9630d14
Zhonghao Wang
698c48066052d3bed9630d15
Yuqian Zhou
698c48066052d3bed9630d16
Zhe Lin
698c48066052d3bed9630d17
Dmitry Baranchuk
Rethinking Global Text Conditioning in Diffusion Transformers
Diffusion transformers typically incorporate textual information via attention layers and a modulation mechanism using a pooled text embedding. Nevertheless, recent approaches discard modulation-based text conditioning and rely exclusively on attention. In this paper, we address whether modulation-based text conditioning is necessary and whether it can provide any performance advantage. Our analysis shows that, in its conventional usage, the pooled embedding contributes little to overall performance, suggesting that attention alone is generally sufficient for faithfully propagating prompt information. However, we reveal that the pooled embedding can provide significant gains when used from a different perspective-serving as guidance and enabling controllable shifts toward more desirable properties. This approach is training-free, simple to implement, incurs negligible runtime overhead, and can be applied to various diffusion models, bringing improvements across diverse tasks, including text-to-image/video generation and image editing.
Modulation-based text conditioning in diffusion transformers provides performance benefits when used as guidance for controllable generation rather than just as attention mechanisms.
diffusion transformers attention layers modulation mechanism pooled text embedding text-to-image generation image editing controllable generation