InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization
We address the task of multi-view image editing from sparse input views, where the inputs can be seen as a mix of images capturing the scene from different viewpoints. The goal is to modify the scene according to a textual instruction while preserving consistency across all views. Existing methods, based on per-scene neural fields or temporal attention mechanisms, struggle in this setting, often producing artifacts and incoherent edits. We propose InstructMix2Mix (I-Mix2Mix), a framework that distills the editing capabilities of a 2D diffusion model into a pretrained multi-view diffusion model, leveraging its data-driven 3D prior for cross-view consistency. A key contribution is replacing the conventional neural field consolidator in Score Distillation Sampling (SDS) with a multi-view diffusion student, which requires novel adaptations: incremental student updates across timesteps, a specialized teacher noise scheduler to prevent degeneration, and an attention modification that enhances cross-view coherence without additional cost. Experiments demonstrate that I-Mix2Mix significantly improves multi-view consistency while maintaining high per-frame edit quality.
69217a26b5612535ed955689
Daniel Gilo
claimed_verified
2025-11-24T07:57:59.618Z
672b303918685a38713dd8f1
/avatars/800802127af17be6370bf7f65b4a0cc9.svg
Daniel Gilo
danielgilo
user
danielgilo
69217a26b5612535ed95568a
Or Litany
InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization
We address the task of multi-view image editing from sparse input views, where the inputs can be seen as a mix of images capturing the scene from different viewpoints. The goal is to modify the scene according to a textual instruction while preserving consistency across all views. Existing methods, based on per-scene neural fields or temporal attention mechanisms, struggle in this setting, often producing artifacts and incoherent edits. We propose InstructMix2Mix (I-Mix2Mix), a framework that distills the editing capabilities of a 2D diffusion model into a pretrained multi-view diffusion model, leveraging its data-driven 3D prior for cross-view consistency. A key contribution is replacing the conventional neural field consolidator in Score Distillation Sampling (SDS) with a multi-view diffusion student, which requires novel adaptations: incremental student updates across timesteps, a specialized teacher noise scheduler to prevent degeneration, and an attention modification that enhances cross-view coherence without additional cost. Experiments demonstrate that I-Mix2Mix significantly improves multi-view consistency while maintaining high per-frame edit quality.
A framework called InstructMix2Mix uses a 2D diffusion model to improve multi-view image editing by leveraging a 3D prior for cross-view consistency.
multi-view image editing textual instruction neural fields temporal attention mechanisms diffusion model 2D diffusion model multi-view diffusion model 3D prior Score Distillation Sampling SDS student updates teacher noise scheduler attention modification cross-view coherence