PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders
Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.
69c67e76878ef8af32384e2a
Niccolò Cavagnero
claimed_verified
2026-03-27T14:56:43.158Z
67ea650e28179c61bef30406
/avatars/bb142e1a8abf29be7d04bcbdf4b2dab5.svg
Niccolò Cavagnero
neikos00
user
neikos00
69c67e76878ef8af32384e2b
Narges Norouzi
69c67e76878ef8af32384e2c
Gijs Dubbelman
69c67e76878ef8af32384e2d
Daan de Geus
PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders
Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.
Plain Mask Decoder enables fast, frozen Vision Foundation Model segmentation with competitive accuracy across image and video tasks.
Vision Foundation Models encoder-only models image segmentation video segmentation Transformer-based decoder frozen VFM features Plain Mask Decoder Plain Mask Transformer