Towards Scalable Pre-training of Visual Tokenizers for Generation
The quality of the latent space in visual tokenizers (e.g., VAEs) is crucial for modern generative models. However, the standard reconstruction-based training paradigm produces a latent space that is biased towards low-level information, leading to a foundation flaw: better pixel-level accuracy does not lead to higher-quality generation. This implies that pouring extensive compute into visual tokenizer pre-training translates poorly to improved performance in generation. We identify this as the ``pre-training scaling problem`` and suggest a necessary shift: to be effective for generation, a latent space must concisely represent high-level semantics. We present VTP, a unified visual tokenizer pre-training framework, pioneering the joint optimization of image-text contrastive, self-supervised, and reconstruction losses. Our large-scale study reveals two principal findings: (1) understanding is a key driver of generation, and (2) much better scaling properties, where generative performance scales effectively with compute, parameters, and data allocated to the pretraining of the visual tokenizer. After large-scale pre-training, our tokenizer delivers a competitive profile (78.2 zero-shot accuracy and 0.36 rFID on ImageNet) and 4.1 times faster convergence on generation compared to advanced distillation methods. More importantly, it scales effectively: without modifying standard DiT training specs, solely investing more FLOPS in pretraining VTP achieves 65.8\% FID improvement in downstream generation, while conventional autoencoder stagnates very early at 1/10 FLOPS. Our pre-trained models are available at https://github.com/MiniMax-AI/VTP.
6940ee0665f1e24a1178066d
Jingfeng Yao
admin_assigned
2025-12-16T10:34:03.365Z
67756c9c846a267749304255
/avatars/01f09805b561887c55d1b9ad4e96b461.svg
Jingfeng Yao
MapleF9
user
MapleF9
6940ee0665f1e24a1178066e
Yuda Song
claimed_verified
2025-12-16T14:10:38.071Z
6264bf5a1ed8d81e47ae3a62
https://cdn-avatars.huggingface.co/v1/production/uploads/1650769741842-noauth.jpeg
Yuda Song
IDKiro
user
IDKiro
6940ee0665f1e24a1178066f
Yucong Zhou
admin_assigned
2025-12-16T10:34:50.881Z
64192280d459c9e7fbb03aa1
/avatars/89935de92f3f9107d7b768b82fb27e70.svg
Zhou
Yucong
user
Yucong
6940ee0665f1e24a11780670
Xinggang Wang
admin_assigned
2025-12-16T10:34:40.199Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
xinggangw
user
xinggangw
Towards Scalable Pre-training of Visual Tokenizers for Generation
The quality of the latent space in visual tokenizers (e.g., VAEs) is crucial for modern generative models. However, the standard reconstruction-based training paradigm produces a latent space that is biased towards low-level information, leading to a foundation flaw: better pixel-level accuracy does not lead to higher-quality generation. This implies that pouring extensive compute into visual tokenizer pre-training translates poorly to improved performance in generation. We identify this as the ``pre-training scaling problem`` and suggest a necessary shift: to be effective for generation, a latent space must concisely represent high-level semantics. We present VTP, a unified visual tokenizer pre-training framework, pioneering the joint optimization of image-text contrastive, self-supervised, and reconstruction losses. Our large-scale study reveals two principal findings: (1) understanding is a key driver of generation, and (2) much better scaling properties, where generative performance scales effectively with compute, parameters, and data allocated to the pretraining of the visual tokenizer. After large-scale pre-training, our tokenizer delivers a competitive profile (78.2 zero-shot accuracy and 0.36 rFID on ImageNet) and 4.1 times faster convergence on generation compared to advanced distillation methods. More importantly, it scales effectively: without modifying standard DiT training specs, solely investing more FLOPS in pretraining VTP achieves 65.8\% FID improvement in downstream generation, while conventional autoencoder stagnates very early at 1/10 FLOPS. Our pre-trained models are available at https://github.com/MiniMax-AI/VTP.
A unified visual tokenizer pre-training framework (VTP) improves generative performance by optimizing image-text contrastive, self-supervised, and reconstruction losses, leading to better scaling properties and higher zero-shot accuracy and faster convergence.
latent space visual tokenizers VAEs reconstruction-based training low-level information pre-training scaling problem high-level semantics VTP image-text contrastive self-supervised reconstruction losses generative performance ImageNet zero-shot accuracy rFID FLOPS DiT advanced distillation methods