Visual Generation Tuning
Large Vision Language Models (VLMs) effectively bridge the modality gap through extensive pretraining, acquiring sophisticated visual representations aligned with language. However, it remains underexplored whether these representations, optimized for multimodal understanding tasks, harbor an inherent potential for visual generation. In this paper, we propose VGT, Visual Generation Tuning, a novel paradigm designed to stimulate the underlying capabilities of visual generation within any vision language models. By performing efficient visual generation tuning on well-pretrained VLMs, we significantly mitigate the alignment costs and accelerate the convergence of autoregressive modeling in the continuous space (20x speedup). Specifically, we dismiss the entangled pixel-level VAEs designed for diffusion transformers and formulate VGT-AE through aligning the semantic encoders from pretrained VLMs with the latent representations of pixel decoders. In image reconstruction tasks, we achieve 26.67 PSNR and 0.50 rFID at a 28x compression ratio, outperforming specialized VAEs; in visual generation tasks, we achieve state-of-the-art outcomes among autoregressive models, 0.77 on GenEval and 78.73 on DPG-Bench. Furthermore, our proposed VGT showcases significant scaling promise and is versatile for endowing any VLMs trained for multimodal understanding with the capabilities of visual generation, which paves the new avenue to explore next-generation unified multimodal foundation models. Models and codes are available at https://github.com/hustvl/VGT.
692e58e037312eaa83fd877c
Jiahao Guo
692e58e037312eaa83fd877d
Sinan Du
692e58e037312eaa83fd877e
Jingfeng Yao
692e58e037312eaa83fd877f
Wenyu Liu
692e58e037312eaa83fd8780
Bo Li
claimed_verified
2025-12-10T09:10:33.793Z
68fed13597cae9c0f484ebed
/avatars/8df380d81daab4fae8a52d0816bf8099.svg
libo
libo31
user
libo31
692e58e037312eaa83fd8781
Haoxiang Cao
692e58e037312eaa83fd8782
Kun Gai
692e58e037312eaa83fd8783
Chun Yuan
692e58e037312eaa83fd8784
Kai Wu
claimed_verified
2025-12-10T09:10:35.705Z
68e741ea3edb0ff47e20084e
https://cdn-avatars.huggingface.co/v1/production/uploads/68e741ea3edb0ff47e20084e/OyBgFqcU4QWyPF_K58Gt5.jpeg
Wu Kai
KaiiWuu1993
user
KaiiWuu1993
692e58e037312eaa83fd8785
Xinggang Wang
Visual Generation Tuning
Large Vision Language Models (VLMs) effectively bridge the modality gap through extensive pretraining, acquiring sophisticated visual representations aligned with language. However, it remains underexplored whether these representations, optimized for multimodal understanding tasks, harbor an inherent potential for visual generation. In this paper, we propose VGT, Visual Generation Tuning, a novel paradigm designed to stimulate the underlying capabilities of visual generation within any vision language models. By performing efficient visual generation tuning on well-pretrained VLMs, we significantly mitigate the alignment costs and accelerate the convergence of autoregressive modeling in the continuous space (20x speedup). Specifically, we dismiss the entangled pixel-level VAEs designed for diffusion transformers and formulate VGT-AE through aligning the semantic encoders from pretrained VLMs with the latent representations of pixel decoders. In image reconstruction tasks, we achieve 26.67 PSNR and 0.50 rFID at a 28x compression ratio, outperforming specialized VAEs; in visual generation tasks, we achieve state-of-the-art outcomes among autoregressive models, 0.77 on GenEval and 78.73 on DPG-Bench. Furthermore, our proposed VGT showcases significant scaling promise and is versatile for endowing any VLMs trained for multimodal understanding with the capabilities of visual generation, which paves the new avenue to explore next-generation unified multimodal foundation models. Models and codes are available at https://github.com/hustvl/VGT.
VGT, a novel paradigm for visual generation tuning, enhances vision language models to achieve high-quality image reconstruction and generation with improved efficiency.
Large Vision Language Models (VLMs) multimodal understanding visual generation VGT Visual Generation Tuning autoregressive modeling continuous space pixel-level VAEs diffusion transformers VGT-AE semantic encoders latent representations pixel decoders image reconstruction PSNR rFID visual generation GenEval DPG-Bench unified multimodal foundation models