Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion
  Models
Text-to-image diffusion models deliver high-quality images, yet aligning them
with human preferences remains challenging. We revisit diffusion-based Direct
Preference Optimization (DPO) for these models and identify a critical
pathology: enlarging the preference margin does not necessarily improve
generation quality. In particular, the standard Diffusion-DPO objective can
increase the reconstruction error of both winner and loser branches.
Consequently, degradation of the less-preferred outputs can become sufficiently
severe that the preferred branch is also adversely affected even as the margin
grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule
that preserves the winner by adaptively scaling the loser gradient according to
its alignment with the winner gradient. A first-order analysis yields a
closed-form scaling coefficient that guarantees the error of the preferred
output is non-increasing at each optimization step. Our method is simple,
model-agnostic, broadly compatible with existing DPO-style alignment frameworks
and adds only marginal computational overhead. Across standard text-to-image
benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning
baselines on automated preference, aesthetic, and prompt alignment metrics.
Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.
6912d73fa644ba07c499c7fa
Minghao Fu
claimed_verified
2025-12-04T08:51:13.134Z
65bb691d00a0399784b5d03d
/avatars/872b5b68a7580e47e42c28e724cff7ce.svg
Minghao Fu
heekhero
user
heekhero
6912d73fa644ba07c499c7fb
Guo-Hua Wang
claimed_verified
2025-11-11T19:43:26.420Z
636f4c6b5d2050767e4a1491
https://cdn-avatars.huggingface.co/v1/production/uploads/636f4c6b5d2050767e4a1491/qErBLQnYzL35EkbDAhvqN.jpeg
Guo-Hua Wang
Flourish
user
Flourish
6912d73fa644ba07c499c7fc
Tianyu Cui
claimed_verified
2025-12-04T08:51:15.101Z
66bc518c47a442c77238f15b
/avatars/e35fec526c0e74cfa20ecdfd74f0ef1f.svg
TianyuCui
Rebelliousgang
user
Rebelliousgang
6912d73fa644ba07c499c7fd
Qing-Guo Chen
6912d73fa644ba07c499c7fe
Zhao Xu
6912d73fa644ba07c499c7ff
Weihua Luo
6912d73fa644ba07c499c800
Kaifu Zhang
Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion
  Models
Text-to-image diffusion models deliver high-quality images, yet aligning them
with human preferences remains challenging. We revisit diffusion-based Direct
Preference Optimization (DPO) for these models and identify a critical
pathology: enlarging the preference margin does not necessarily improve
generation quality. In particular, the standard Diffusion-DPO objective can
increase the reconstruction error of both winner and loser branches.
Consequently, degradation of the less-preferred outputs can become sufficiently
severe that the preferred branch is also adversely affected even as the margin
grows. To address this, we introduce Diffusion-SDPO, a safeguarded update rule
that preserves the winner by adaptively scaling the loser gradient according to
its alignment with the winner gradient. A first-order analysis yields a
closed-form scaling coefficient that guarantees the error of the preferred
output is non-increasing at each optimization step. Our method is simple,
model-agnostic, broadly compatible with existing DPO-style alignment frameworks
and adds only marginal computational overhead. Across standard text-to-image
benchmarks, Diffusion-SDPO delivers consistent gains over preference-learning
baselines on automated preference, aesthetic, and prompt alignment metrics.
Code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO.
Diffusion-SDPO improves text-to-image generation quality by adaptively scaling the loser gradient in preference optimization, ensuring the preferred output's error does not increase.
diffusion models Direct Preference Optimization (DPO) preference margin reconstruction error winner branch loser branch Diffusion-SDPO safeguarded update rule adaptive scaling winner gradient loser gradient alignment closed-form scaling coefficient automated preference aesthetic prompt alignment metrics