CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization
Autoregressive (AR) language models rely on causal tokenization, but extending this paradigm to vision remains non-trivial. Current visual tokenizers either flatten 2D patches into non-causal sequences or enforce heuristic orderings that misalign with the "next-token prediction" pattern. Recent diffusion autoencoders similarly fall short: conditioning the decoder on all tokens lacks causality, while applying nested dropout mechanism introduces imbalance. To address these challenges, we present CaTok, a 1D causal image tokenizer with a MeanFlow decoder. By selecting tokens over time intervals and binding them to the MeanFlow objective, as illustrated in Fig. 1, CaTok learns causal 1D representations that support both fast one-step generation and high-fidelity multi-step sampling, while naturally capturing diverse visual concepts across token intervals. To further stabilize and accelerate training, we propose a straightforward regularization REPA-A, which aligns encoder features with Vision Foundation Models (VFMs). Experiments demonstrate that CaTok achieves state-of-the-art results on ImageNet reconstruction, reaching 0.75 FID, 22.53 PSNR and 0.674 SSIM with fewer training epochs, and the AR model attains performance comparable to leading approaches.
69afa6d9b3ab31dee1281e87
Yitong Chen
claimed_verified
2026-03-10T09:44:37.061Z
64651db3611ae99d14d392ea
/avatars/b818dc0dddc999758ab5737d5053e8c3.svg
YitongChen (SII)
Row11n
user
Row11n
69afa6d9b3ab31dee1281e88
Zuxuan Wu
69afa6d9b3ab31dee1281e89
Xipeng Qiu
69afa6d9b3ab31dee1281e8a
Yu-Gang Jiang
CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization
Autoregressive (AR) language models rely on causal tokenization, but extending this paradigm to vision remains non-trivial. Current visual tokenizers either flatten 2D patches into non-causal sequences or enforce heuristic orderings that misalign with the "next-token prediction" pattern. Recent diffusion autoencoders similarly fall short: conditioning the decoder on all tokens lacks causality, while applying nested dropout mechanism introduces imbalance. To address these challenges, we present CaTok, a 1D causal image tokenizer with a MeanFlow decoder. By selecting tokens over time intervals and binding them to the MeanFlow objective, as illustrated in Fig. 1, CaTok learns causal 1D representations that support both fast one-step generation and high-fidelity multi-step sampling, while naturally capturing diverse visual concepts across token intervals. To further stabilize and accelerate training, we propose a straightforward regularization REPA-A, which aligns encoder features with Vision Foundation Models (VFMs). Experiments demonstrate that CaTok achieves state-of-the-art results on ImageNet reconstruction, reaching 0.75 FID, 22.53 PSNR and 0.674 SSIM with fewer training epochs, and the AR model attains performance comparable to leading approaches.
CaTok presents a 1D causal image tokenizer with a MeanFlow decoder that enables fast one-step generation and high-fidelity multi-step sampling while achieving state-of-the-art image reconstruction performance.
autoregressive language models causal tokenization visual tokenizers diffusion autoencoders MeanFlow decoder REPA-A Vision Foundation Models FID PSNR SSIM