Communication-Inspired Tokenization for Structured Image Representations
Discrete image tokenizers have emerged as a key component of modern vision and multimodal systems, providing a sequential interface for transformer-based architectures. However, most existing approaches remain primarily optimized for reconstruction and compression, often yielding tokens that capture local texture rather than object-level semantic structure. Inspired by the incremental and compositional nature of human communication, we introduce COMmunication inspired Tokenization (COMiT), a framework for learning structured discrete visual token sequences. COMiT constructs a latent message within a fixed token budget by iteratively observing localized image crops and recurrently updating its discrete representation. At each step, the model integrates new visual information while refining and reorganizing the existing token sequence. After several encoding iterations, the final message conditions a flow-matching decoder that reconstructs the full image. Both encoding and decoding are implemented within a single transformer model and trained end-to-end using a combination of flow-matching reconstruction and semantic representation alignment losses. Our experiments demonstrate that while semantic alignment provides grounding, attentive sequential tokenization is critical for inducing interpretable, object-centric token structure and substantially improving compositional generalization and relational reasoning over prior methods.
699eb3f5dfbcf0b800aecc92
Aram Davtyan
699eb3f5dfbcf0b800aecc93
Yusuf Sahin
699eb3f5dfbcf0b800aecc94
Yasaman Haghighi
699eb3f5dfbcf0b800aecc95
Sebastian Stapf
699eb3f5dfbcf0b800aecc96
Pablo Acuaviva
699eb3f5dfbcf0b800aecc97
Alexandre Alahi
699eb3f5dfbcf0b800aecc98
Paolo Favaro
Communication-Inspired Tokenization for Structured Image Representations
Discrete image tokenizers have emerged as a key component of modern vision and multimodal systems, providing a sequential interface for transformer-based architectures. However, most existing approaches remain primarily optimized for reconstruction and compression, often yielding tokens that capture local texture rather than object-level semantic structure. Inspired by the incremental and compositional nature of human communication, we introduce COMmunication inspired Tokenization (COMiT), a framework for learning structured discrete visual token sequences. COMiT constructs a latent message within a fixed token budget by iteratively observing localized image crops and recurrently updating its discrete representation. At each step, the model integrates new visual information while refining and reorganizing the existing token sequence. After several encoding iterations, the final message conditions a flow-matching decoder that reconstructs the full image. Both encoding and decoding are implemented within a single transformer model and trained end-to-end using a combination of flow-matching reconstruction and semantic representation alignment losses. Our experiments demonstrate that while semantic alignment provides grounding, attentive sequential tokenization is critical for inducing interpretable, object-centric token structure and substantially improving compositional generalization and relational reasoning over prior methods.
COMiT framework learns structured discrete visual tokens through iterative encoding and flow-matching decoding, improving object-centric representation and compositional generalization.
discrete image tokenizers vision transformers flow-matching semantic representation alignment attentive sequential tokenization object-centric token structure compositional generalization relational reasoning latent message transformer model