VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
Unifying multimodal understanding, generation and reconstruction representation in a single tokenizer remains a key challenge in building unified models. Previous research predominantly attempts to address this in a dual encoder paradigm, e.g., utilizing the separate encoders for understanding and generation respectively or balancing semantic representations and low-level features with contrastive loss. In this paper, we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer. Specifically, we build upon pretrained vision foundation models with a symmetric ViT decoder and adopt a two-stage training strategy: first, it freezes the encoder and learns a high-dimensional semantic VQ codebook with pixel reconstruction objective; then jointly optimizes the encoder with self-distillation constraints. This design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens that are compatible for generation and fine-grained reconstruction. Besides, we identify the intriguing property in quantizing semantic encoders that rely on high-dimensional codebook in contrast to the previous common practice of low-dimensional codebook in image reconstruction. The semantic VQ codebook can achieve a 100% utilization ratio at a dimension of 1536. VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction with promising scaling property in the autoregressive paradigm for its discrete merits.
693ba6879874a2a5e4ffb362
Sinan Du
693ba6879874a2a5e4ffb363
Jiahao Guo
693ba6879874a2a5e4ffb364
Bo Li
claimed_verified
2025-12-12T09:14:49.405Z
68fed13597cae9c0f484ebed
/avatars/8df380d81daab4fae8a52d0816bf8099.svg
libo
libo31
user
libo31
693ba6879874a2a5e4ffb365
Shuhao Cui
693ba6879874a2a5e4ffb366
Zhengzhuo Xu
693ba6879874a2a5e4ffb367
Yifu Luo
693ba6879874a2a5e4ffb368
Yongxian Wei
693ba6879874a2a5e4ffb369
Kun Gai
693ba6879874a2a5e4ffb36a
Xinggang Wang
693ba6879874a2a5e4ffb36b
Kai Wu
claimed_verified
2025-12-12T09:14:47.442Z
68e741ea3edb0ff47e20084e
https://cdn-avatars.huggingface.co/v1/production/uploads/68e741ea3edb0ff47e20084e/OyBgFqcU4QWyPF_K58Gt5.jpeg
Wu Kai
KaiiWuu1993
user
KaiiWuu1993
693ba6879874a2a5e4ffb36c
Chun Yuan
VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
Unifying multimodal understanding, generation and reconstruction representation in a single tokenizer remains a key challenge in building unified models. Previous research predominantly attempts to address this in a dual encoder paradigm, e.g., utilizing the separate encoders for understanding and generation respectively or balancing semantic representations and low-level features with contrastive loss. In this paper, we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer. Specifically, we build upon pretrained vision foundation models with a symmetric ViT decoder and adopt a two-stage training strategy: first, it freezes the encoder and learns a high-dimensional semantic VQ codebook with pixel reconstruction objective; then jointly optimizes the encoder with self-distillation constraints. This design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens that are compatible for generation and fine-grained reconstruction. Besides, we identify the intriguing property in quantizing semantic encoders that rely on high-dimensional codebook in contrast to the previous common practice of low-dimensional codebook in image reconstruction. The semantic VQ codebook can achieve a 100% utilization ratio at a dimension of 1536. VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction with promising scaling property in the autoregressive paradigm for its discrete merits.
VQRAE, a Vector Quantization Representation AutoEncoder, unifies multimodal understanding, generation, and reconstruction using a unified tokenizer with continuous semantic features and discrete tokens.
multimodal understanding generation reconstruction tokenizer dual encoder paradigm VQRAE Representation AutoEncoders Vector Quantization symmetric ViT decoder two-stage training strategy encoder self-distillation constraints semantic VQ codebook pixel reconstruction objective multimodal understanding discrete tokens fine-grained reconstruction autoregressive paradigm