S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation
In recent years, the state-of-the-art in unsupervised video instance segmentation has heavily relied on synthetic video data, generated from object-centric image datasets such as ImageNet. However, video synthesis by artificially shifting and scaling image instance masks fails to accurately model realistic motion in videos, such as perspective changes, movement by parts of one or multiple instances, or camera motion. To tackle this issue, we propose an unsupervised video instance segmentation model trained exclusively on real video data. We start from unsupervised instance segmentation masks on individual video frames. However, these single-frame segmentations exhibit temporal noise and their quality varies through the video. Therefore, we establish temporal coherence by identifying high-quality keymasks in the video by leveraging deep motion priors. The sparse keymask pseudo-annotations are then used to train a segmentation model for implicit mask propagation, for which we propose a Sparse-To-Dense Distillation approach aided by a Temporal DropLoss. After training the final model on the resulting dense labelset, our approach outperforms the current state-of-the-art across various benchmarks.
694249cb5d5b2dc105274946
Leon Sick
claimed_verified
2025-12-17T10:07:49.404Z
642150c201c62c1e41fc2390
https://cdn-avatars.huggingface.co/v1/production/uploads/642150c201c62c1e41fc2390/4udBai02RARpIujIeiCX-.png
Leon Sick
leonsick
user
leonsick
694249cb5d5b2dc105274947
Lukas Hoyer
admin_assigned
2025-12-17T10:12:13.029Z
6583f5131804c2d060b90ee3
/avatars/6c50794d303174d4a5cd04301a45a3ca.svg
Lukas Hoyer
lhoyer
user
lhoyer
694249cb5d5b2dc105274948
Dominik Engel
admin_assigned
2025-12-17T10:12:18.622Z
6424275e41289875f61f68e4
/avatars/b1582cabc83711df249a41bafaadbfc2.svg
Dominik Engel
xeTaiz
user
xeTaiz
694249cb5d5b2dc105274949
Pedro Hermosilla
admin_assigned
2025-12-17T10:12:23.642Z
67a37342e7acfe540dcd5f0e
/avatars/759a4769ab390072e62859f5c1db3c0a.svg
Pedro Hermosilla
phermosilla
user
phermosilla
694249cb5d5b2dc10527494a
Timo Ropinski
S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation
In recent years, the state-of-the-art in unsupervised video instance segmentation has heavily relied on synthetic video data, generated from object-centric image datasets such as ImageNet. However, video synthesis by artificially shifting and scaling image instance masks fails to accurately model realistic motion in videos, such as perspective changes, movement by parts of one or multiple instances, or camera motion. To tackle this issue, we propose an unsupervised video instance segmentation model trained exclusively on real video data. We start from unsupervised instance segmentation masks on individual video frames. However, these single-frame segmentations exhibit temporal noise and their quality varies through the video. Therefore, we establish temporal coherence by identifying high-quality keymasks in the video by leveraging deep motion priors. The sparse keymask pseudo-annotations are then used to train a segmentation model for implicit mask propagation, for which we propose a Sparse-To-Dense Distillation approach aided by a Temporal DropLoss. After training the final model on the resulting dense labelset, our approach outperforms the current state-of-the-art across various benchmarks.
An unsupervised video instance segmentation model using real video data and deep motion priors outperforms existing methods by establishing temporal coherence and using sparse-to-dense distillation.
unsupervised video instance segmentation synthetic video data object-centric image datasets deep motion priors keymasks implicit mask propagation Sparse-To-Dense Distillation Temporal DropLoss