Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing
Modern Latent Diffusion Models (LDMs) typically operate in low-level Variational Autoencoder (VAE) latent spaces that are primarily optimized for pixel-level reconstruction. To unify vision generation and understanding, a burgeoning trend is to adopt high-dimensional features from representation encoders as generative latents. However, we empirically identify two fundamental obstacles in this paradigm: (1) the discriminative feature space lacks compact regularization, making diffusion models prone to off-manifold latents that lead to inaccurate object structures; and (2) the encoder's inherently weak pixel-level reconstruction hinders the generator from learning accurate fine-grained geometry and texture. In this paper, we propose a systematic framework to adapt understanding-oriented encoder features for generative tasks. We introduce a semantic-pixel reconstruction objective to regularize the latent space, enabling the compression of both semantic information and fine-grained details into a highly compact representation (96 channels with 16x16 spatial downsampling). This design ensures that the latent space remains semantically rich and achieves state-of-the-art image reconstruction, while remaining compact enough for accurate generation. Leveraging this representation, we design a unified Text-to-Image (T2I) and image editing model. Benchmarking against various feature spaces, we demonstrate that our approach achieves state-of-the-art reconstruction, faster convergence, and substantial performance gains in both T2I and editing tasks, validating that representation encoders can be effectively adapted into robust generative components.
6948d11034f46eaf46cbb338
Shilong Zhang
6948d11034f46eaf46cbb339
He Zhang
6948d11034f46eaf46cbb33a
Zhifei Zhang
6948d11034f46eaf46cbb33b
Chongjian Ge
6948d11034f46eaf46cbb33c
Shuchen Xue
6948d11034f46eaf46cbb33d
Shaoteng Liu
6948d11034f46eaf46cbb33e
Mengwei Ren
6948d11034f46eaf46cbb33f
Soo Ye Kim
6948d11034f46eaf46cbb340
Yuqian Zhou
6948d11034f46eaf46cbb341
Qing Liu
6948d11034f46eaf46cbb342
Daniil Pakhomov
6948d11034f46eaf46cbb343
Kai Zhang
6948d11034f46eaf46cbb344
Zhe Lin
6948d11034f46eaf46cbb345
Ping Luo
Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing
Modern Latent Diffusion Models (LDMs) typically operate in low-level Variational Autoencoder (VAE) latent spaces that are primarily optimized for pixel-level reconstruction. To unify vision generation and understanding, a burgeoning trend is to adopt high-dimensional features from representation encoders as generative latents. However, we empirically identify two fundamental obstacles in this paradigm: (1) the discriminative feature space lacks compact regularization, making diffusion models prone to off-manifold latents that lead to inaccurate object structures; and (2) the encoder's inherently weak pixel-level reconstruction hinders the generator from learning accurate fine-grained geometry and texture. In this paper, we propose a systematic framework to adapt understanding-oriented encoder features for generative tasks. We introduce a semantic-pixel reconstruction objective to regularize the latent space, enabling the compression of both semantic information and fine-grained details into a highly compact representation (96 channels with 16x16 spatial downsampling). This design ensures that the latent space remains semantically rich and achieves state-of-the-art image reconstruction, while remaining compact enough for accurate generation. Leveraging this representation, we design a unified Text-to-Image (T2I) and image editing model. Benchmarking against various feature spaces, we demonstrate that our approach achieves state-of-the-art reconstruction, faster convergence, and substantial performance gains in both T2I and editing tasks, validating that representation encoders can be effectively adapted into robust generative components.
Latent diffusion models using representation encoder features face challenges in semantic compactness and pixel-level reconstruction, which are addressed through a semantic-pixel reconstruction objective that enables compact yet semantically rich representations for unified text-to-image and image editing tasks.
latent diffusion models variational autoencoder representation encoders generative latents semantic-pixel reconstruction off-manifold latents fine-grained geometry texture compact representation text-to-image image editing convergence state-of-the-art reconstruction