VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x--10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/
699c50c4cf1450b05134decd
Narges Norouzi
699c50c4cf1450b05134dece
Idil Esen Zulfikar
699c50c4cf1450b05134decf
Niccol`o Cavagnero
claimed_verified
2026-02-25T17:32:11.289Z
67ea650e28179c61bef30406
/avatars/bb142e1a8abf29be7d04bcbdf4b2dab5.svg
Niccolò Cavagnero
neikos00
user
neikos00
699c50c4cf1450b05134ded0
Tommie Kerssies
699c50c4cf1450b05134ded1
Bastian Leibe
699c50c4cf1450b05134ded2
Gijs Dubbelman
699c50c4cf1450b05134ded3
Daan de Geus
claimed_verified
2026-02-24T09:51:34.709Z
6392408ae836c1426b776472
/avatars/380b66f142197572e4d8516046c82508.svg
Daan de Geus
ddegeus
user
ddegeus
VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x--10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/
A video segmentation model eliminates specialized tracking modules by using a Vision Transformer encoder with query propagation and fusion mechanisms for efficient, high-speed processing.
Vision Transformer encoder-only video segmentation query propagation query fusion mask transformer temporal modeling frame-by-frame processing