Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models.
6972ea8bfb12c92b735b74a8
Shengbang Tong
6972ea8bfb12c92b735b74a9
Boyang Zheng
6972ea8bfb12c92b735b74aa
Ziteng Wang
claimed_verified
2026-01-23T09:38:01.136Z
64249f76d476e4ad55665d59
https://cdn-avatars.huggingface.co/v1/production/uploads/64249f76d476e4ad55665d59/msLU1k5dcdRuf2FwaWvCR.jpeg
Ziteng Wang
AustinWang0330
user
AustinWang0330
6972ea8bfb12c92b735b74ab
Bingda Tang
6972ea8bfb12c92b735b74ac
Nanye Ma
6972ea8bfb12c92b735b74ad
Ellis Brown
claimed_verified
2026-01-23T20:15:10.637Z
626dc5105f7327906f0b2a4e
https://cdn-avatars.huggingface.co/v1/production/uploads/626dc5105f7327906f0b2a4e/QCSzuwYqsv8ozRnusVb-F.jpeg
Ellis Brown
ellisbrown
user
ellisbrown
6972ea8bfb12c92b735b74ae
Jihan Yang
claimed_verified
2026-01-26T08:32:46.014Z
6304baf041387c7f1177a5d2
https://cdn-avatars.huggingface.co/v1/production/uploads/6304baf041387c7f1177a5d2/cQgCR8AsrMUaF2QVh97I9.jpeg
Jihan Yang
jihanyang
user
jihanyang
6972ea8bfb12c92b735b74af
Rob Fergus
6972ea8bfb12c92b735b74b0
Yann LeCun
6972ea8bfb12c92b735b74b1
Saining Xie
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation. We first scale RAE decoders on the frozen representation encoder (SigLIP-2) beyond ImageNet by training on web, synthetic, and text-rendering data, finding that while scale improves general fidelity, targeted data composition is essential for specific domains like text. We then rigorously stress-test the RAE design choices originally proposed for ImageNet. Our analysis reveals that scaling simplifies the framework: while dimension-dependent noise scheduling remains critical, architectural complexities such as wide diffusion heads and noise-augmented decoding offer negligible benefits at scale Building on this simplified framework, we conduct a controlled comparison of RAE against the state-of-the-art FLUX VAE across diffusion transformer scales from 0.5B to 9.8B parameters. RAEs consistently outperform VAEs during pretraining across all model scales. Further, during finetuning on high-quality datasets, VAE-based models catastrophically overfit after 64 epochs, while RAE models remain stable through 256 epochs and achieve consistently better performance. Across all experiments, RAE-based diffusion models demonstrate faster convergence and better generation quality, establishing RAEs as a simpler and stronger foundation than VAEs for large-scale T2I generation. Additionally, because both visual understanding and generation can operate in a shared representation space, the multimodal model can directly reason over generated latents, opening new possibilities for unified models.
Representation Autoencoders (RAEs) demonstrate superior performance over VAEs in large-scale text-to-image generation, showing improved stability, faster convergence, and better quality while enabling unified multimodal reasoning in shared representation spaces.
representation autoencoders diffusion modeling semantic latent spaces text-to-image generation frozen representation encoder SigLIP-2 noise scheduling diffusion transformers pretraining finetuning catastrophic overfitting multimodal model shared representation space