Vector Quantization using Gaussian Variational Autoencoder
Vector quantized variational autoencoder (VQ-VAE) is a discrete auto-encoder that compresses images into discrete tokens. It is difficult to train due to discretization. In this paper, we propose a simple yet effective technique, dubbed Gaussian Quant (GQ), that converts a Gaussian VAE with certain constraint into a VQ-VAE without training. GQ generates random Gaussian noise as a codebook and finds the closest noise to the posterior mean. Theoretically, we prove that when the logarithm of the codebook size exceeds the bits-back coding rate of the Gaussian VAE, a small quantization error is guaranteed. Practically, we propose a heuristic to train Gaussian VAE for effective GQ, named target divergence constraint (TDC). Empirically, we show that GQ outperforms previous VQ-VAEs, such as VQGAN, FSQ, LFQ, and BSQ, on both UNet and ViT architectures. Furthermore, TDC also improves upon previous Gaussian VAE discretization methods, such as TokenBridge. The source code is provided in https://github.com/tongdaxu/VQ-VAE-from-Gaussian-VAE.
6937a8ab19d912300c34a331
Tongda Xu
admin_assigned
2025-12-09T11:50:56.108Z
64e805afd884b768fc610880
https://cdn-avatars.huggingface.co/v1/production/uploads/64e805afd884b768fc610880/VMfKuaXF6JLp2I2DDv3zH.png
tongda xu
xutongda
user
xutongda
6937a8ab19d912300c34a332
Wendi Zheng
6937a8ab19d912300c34a333
Jiajun He
6937a8ab19d912300c34a334
Jose Miguel Hernandez-Lobato
admin_assigned
2025-12-09T11:51:09.074Z
6874c9f3a2bf37fec355f536
/avatars/eb08589b69c71b32d431c6afe0fe847e.svg
Jose Miguel Hernandez Lobato
jmhernandezlobato
user
jmhernandezlobato
6937a8ab19d912300c34a335
Yan Wang
6937a8ab19d912300c34a336
Ya-Qin Zhang
admin_assigned
2025-12-09T11:51:18.821Z
656c1600665d15428a8bdc60
/avatars/dfdf5f8712c769896b4ab655c4ce32e4.svg
Ya-Qin Zhang
Ya-Qin
user
Ya-Qin
6937a8ab19d912300c34a337
Jie Tang
admin_assigned
2025-12-09T11:51:27.511Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
jerytang
user
jerytang
Vector Quantization using Gaussian Variational Autoencoder
Vector quantized variational autoencoder (VQ-VAE) is a discrete auto-encoder that compresses images into discrete tokens. It is difficult to train due to discretization. In this paper, we propose a simple yet effective technique, dubbed Gaussian Quant (GQ), that converts a Gaussian VAE with certain constraint into a VQ-VAE without training. GQ generates random Gaussian noise as a codebook and finds the closest noise to the posterior mean. Theoretically, we prove that when the logarithm of the codebook size exceeds the bits-back coding rate of the Gaussian VAE, a small quantization error is guaranteed. Practically, we propose a heuristic to train Gaussian VAE for effective GQ, named target divergence constraint (TDC). Empirically, we show that GQ outperforms previous VQ-VAEs, such as VQGAN, FSQ, LFQ, and BSQ, on both UNet and ViT architectures. Furthermore, TDC also improves upon previous Gaussian VAE discretization methods, such as TokenBridge. The source code is provided in https://github.com/tongdaxu/VQ-VAE-from-Gaussian-VAE.
Gaussian Quant (GQ) converts Gaussian VAE to VQ-VAE without training, outperforming previous VQ-VAEs and Gaussian VAE discretization methods across different architectures.
Vector quantized variational autoencoder VQ-VAE Gaussian VAE Gaussian Quant GQ codebook posterior mean bits-back coding rate target divergence constraint TDC VQGAN FSQ LFQ BSQ TokenBridge UNet ViT