GenMask: Adapting DiT for Segmentation via Direct Mask
Recent approaches for segmentation have leveraged pretrained generative models as feature extractors, treating segmentation as a downstream adaptation task via indirect feature retrieval. This implicit use suffers from a fundamental misalignment in representation. It also depends heavily on indirect feature extraction pipelines, which complicate the workflow and limit adaptation. In this paper, we argue that instead of indirect adaptation, segmentation tasks should be trained directly in a generative manner. We identify a key obstacle to this unified formulation: VAE latents of binary masks are sharply distributed, noise robust, and linearly separable, distinct from natural image latents. To bridge this gap, we introduce timesteps sampling strategy for binary masks that emphasizes extreme noise levels for segmentation and moderate noise for image generation, enabling harmonious joint training. We present GenMask, a DiT trains to generate black-and-white segmentation masks as well as colorful images in RGB space under the original generative objective. GenMask preserves the original DiT architecture while removing the need of feature extraction pipelines tailored for segmentation tasks. Empirically, GenMask attains state-of-the-art performance on referring and reasoning segmentation benchmarks and ablations quantify the contribution of each component.
69c4a2034ae7eef815b9d9ec
Yuhuan Yang
claimed_verified
2026-03-27T11:35:04.630Z
6302f300056ec3a2a8754943
/avatars/f111b7061633d3e65dd30967a6b68c96.svg
Yuhuan Yang
yuhuanyang
user
yuhuanyang
69c4a2034ae7eef815b9d9ed
Xianwei Zhuang
69c4a2034ae7eef815b9d9ee
Yuxuan Cai
69c4a2034ae7eef815b9d9ef
Chaofan Ma
claimed_verified
2026-03-27T11:35:06.866Z
63048965eb6d777a838cb7a8
/avatars/b987fb7f630443bf94a03daf8dcbffe9.svg
chaofanma
chaofanma
user
chaofanma
69c4a2034ae7eef815b9d9f0
Shuai Bai
69c4a2034ae7eef815b9d9f1
Jiangchao Yao
69c4a2034ae7eef815b9d9f2
Ya Zhang
69c4a2034ae7eef815b9d9f3
Junyang Lin
69c4a2034ae7eef815b9d9f4
Yanfeng Wang
GenMask: Adapting DiT for Segmentation via Direct Mask
Recent approaches for segmentation have leveraged pretrained generative models as feature extractors, treating segmentation as a downstream adaptation task via indirect feature retrieval. This implicit use suffers from a fundamental misalignment in representation. It also depends heavily on indirect feature extraction pipelines, which complicate the workflow and limit adaptation. In this paper, we argue that instead of indirect adaptation, segmentation tasks should be trained directly in a generative manner. We identify a key obstacle to this unified formulation: VAE latents of binary masks are sharply distributed, noise robust, and linearly separable, distinct from natural image latents. To bridge this gap, we introduce timesteps sampling strategy for binary masks that emphasizes extreme noise levels for segmentation and moderate noise for image generation, enabling harmonious joint training. We present GenMask, a DiT trains to generate black-and-white segmentation masks as well as colorful images in RGB space under the original generative objective. GenMask preserves the original DiT architecture while removing the need of feature extraction pipelines tailored for segmentation tasks. Empirically, GenMask attains state-of-the-art performance on referring and reasoning segmentation benchmarks and ablations quantify the contribution of each component.
Generative models trained directly for segmentation tasks outperform indirect adaptation methods by using a novel timestep sampling strategy that enables joint training for both image generation and binary mask synthesis.
pretrained generative models feature extractors segmentation generative manner VAE latents binary masks DiT timesteps sampling strategy joint training GenMask