UniLumos: Fast and Unified Image and Video Relighting with
  Physics-Plausible Feedback
Relighting is a crucial task with both practical demand and artistic value,
and recent diffusion models have shown strong potential by enabling rich and
controllable lighting effects. However, as they are typically optimized in
semantic latent space, where proximity does not guarantee physical correctness
in visual space, they often produce unrealistic results, such as overexposed
highlights, misaligned shadows, and incorrect occlusions. We address this with
UniLumos, a unified relighting framework for both images and videos that brings
RGB-space geometry feedback into a flow matching backbone. By supervising the
model with depth and normal maps extracted from its outputs, we explicitly
align lighting effects with the scene structure, enhancing physical
plausibility. Nevertheless, this feedback requires high-quality outputs for
supervision in visual space, making standard multi-step denoising
computationally expensive. To mitigate this, we employ path consistency
learning, allowing supervision to remain effective even under few-step training
regimes. To enable fine-grained relighting control and supervision, we design a
structured six-dimensional annotation protocol capturing core illumination
attributes. Building upon this, we propose LumosBench, a disentangled
attribute-level benchmark that evaluates lighting controllability via large
vision-language models, enabling automatic and interpretable assessment of
relighting precision across individual dimensions. Extensive experiments
demonstrate that UniLumos achieves state-of-the-art relighting quality with
significantly improved physical consistency, while delivering a 20x speedup for
both image and video relighting. Code is available at
https://github.com/alibaba-damo-academy/Lumos-Custom.
6909a211812eca10f9cc63ca
Ropeway Liu
6909a211812eca10f9cc63cb
Hangjie Yuan
6909a211812eca10f9cc63cc
Bo Dong
6909a211812eca10f9cc63cd
Jiazheng Xing
6909a211812eca10f9cc63ce
Jinwang Wang
6909a211812eca10f9cc63cf
Rui Zhao
claimed_verified
2025-11-04T09:24:59.454Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
ruizhaocv
user
ruizhaocv
6909a211812eca10f9cc63d0
Yan Xing
6909a211812eca10f9cc63d1
Weihua Chen
6909a211812eca10f9cc63d2
Fan Wang
UniLumos: Fast and Unified Image and Video Relighting with
  Physics-Plausible Feedback
Relighting is a crucial task with both practical demand and artistic value,
and recent diffusion models have shown strong potential by enabling rich and
controllable lighting effects. However, as they are typically optimized in
semantic latent space, where proximity does not guarantee physical correctness
in visual space, they often produce unrealistic results, such as overexposed
highlights, misaligned shadows, and incorrect occlusions. We address this with
UniLumos, a unified relighting framework for both images and videos that brings
RGB-space geometry feedback into a flow matching backbone. By supervising the
model with depth and normal maps extracted from its outputs, we explicitly
align lighting effects with the scene structure, enhancing physical
plausibility. Nevertheless, this feedback requires high-quality outputs for
supervision in visual space, making standard multi-step denoising
computationally expensive. To mitigate this, we employ path consistency
learning, allowing supervision to remain effective even under few-step training
regimes. To enable fine-grained relighting control and supervision, we design a
structured six-dimensional annotation protocol capturing core illumination
attributes. Building upon this, we propose LumosBench, a disentangled
attribute-level benchmark that evaluates lighting controllability via large
vision-language models, enabling automatic and interpretable assessment of
relighting precision across individual dimensions. Extensive experiments
demonstrate that UniLumos achieves state-of-the-art relighting quality with
significantly improved physical consistency, while delivering a 20x speedup for
both image and video relighting. Code is available at
https://github.com/alibaba-damo-academy/Lumos-Custom.
UniLumos, a unified relighting framework, enhances physical plausibility by integrating RGB-space geometry feedback into a flow matching backbone, achieving state-of-the-art results with improved consistency and speed.
diffusion models semantic latent space RGB-space flow matching depth maps normal maps path consistency learning six-dimensional annotation protocol disentangled attribute-level benchmark LumosBench large vision-language models relighting quality physical consistency