Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows
We introduce Iwin Transformer, a novel position-embedding-free hierarchical vision transformer, which can be fine-tuned directly from low to high resolution, through the collaboration of innovative interleaved window attention and depthwise separable convolution. This approach uses attention to connect distant tokens and applies convolution to link neighboring tokens, enabling global information exchange within a single module, overcoming Swin Transformer's limitation of requiring two consecutive blocks to approximate global attention. Extensive experiments on visual benchmarks demonstrate that Iwin Transformer exhibits strong competitiveness in tasks such as image classification (87.4 top-1 accuracy on ImageNet-1K), semantic segmentation and video action recognition. We also validate the effectiveness of the core component in Iwin as a standalone module that can seamlessly replace the self-attention module in class-conditional image generation. The concepts and methods introduced by the Iwin Transformer have the potential to inspire future research, like Iwin 3D Attention in video generation. The code and models are available at https://github.com/cominder/Iwin-Transformer.
6883591b6d99b948a364d1bc
Simin Huo
admin_assigned
2025-07-25T15:09:55.852Z
6562baa7516893ddeb9ad518
https://cdn-avatars.huggingface.co/v1/production/uploads/6562baa7516893ddeb9ad518/xHyyR-F-x3UxTr3Zw2pIj.jpeg
sameen
cominder
user
cominder
6883591b6d99b948a364d1bd
Ning Li
Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows
We introduce Iwin Transformer, a novel position-embedding-free hierarchical vision transformer, which can be fine-tuned directly from low to high resolution, through the collaboration of innovative interleaved window attention and depthwise separable convolution. This approach uses attention to connect distant tokens and applies convolution to link neighboring tokens, enabling global information exchange within a single module, overcoming Swin Transformer's limitation of requiring two consecutive blocks to approximate global attention. Extensive experiments on visual benchmarks demonstrate that Iwin Transformer exhibits strong competitiveness in tasks such as image classification (87.4 top-1 accuracy on ImageNet-1K), semantic segmentation and video action recognition. We also validate the effectiveness of the core component in Iwin as a standalone module that can seamlessly replace the self-attention module in class-conditional image generation. The concepts and methods introduced by the Iwin Transformer have the potential to inspire future research, like Iwin 3D Attention in video generation. The code and models are available at https://github.com/cominder/Iwin-Transformer.
Iwin Transformer is a position-embedding-free hierarchical vision transformer that enables direct fine-tuning across resolutions through interleaved window attention and depthwise separable convolution, demonstrating strong performance in image classification, semantic segmentation, and video action recognition.
position-embedding-free hierarchical vision transformer interleaved window attention depthwise separable convolution self-attention module image classification semantic segmentation video action recognition