Learning Latent Proxies for Controllable Single-Image Relighting
Single-image relighting is highly under-constrained: small illumination changes can produce large, nonlinear variations in shading, shadows, and specularities, while geometry and materials remain unobserved. Existing diffusion-based approaches either rely on intrinsic or G-buffer pipelines that require dense and fragile supervision, or operate purely in latent space without physical grounding, making fine-grained control of direction, intensity, and color unreliable. We observe that a full intrinsic decomposition is unnecessary and redundant for accurate relighting. Instead, sparse but physically meaningful cues, indicating where illumination should change and how materials should respond, are sufficient to guide a diffusion model. Based on this insight, we introduce LightCtrl that integrates physical priors at two levels: a few-shot latent proxy encoder that extracts compact material-geometry cues from limited PBR supervision, and a lighting-aware mask that identifies sensitive illumination regions and steers the denoiser toward shading relevant pixels. To compensate for scarce PBR data, we refine the proxy branch using a DPO-based objective that enforces physical consistency in the predicted cues. We also present ScaLight, a large-scale object-level dataset with systematically varied illumination and complete camera-light metadata, enabling physically consistent and controllable training. Across object and scene level benchmarks, our method achieves photometrically faithful relighting with accurate continuous control, surpassing prior diffusion and intrinsic-based baselines, including gains of up to +2.4 dB PSNR and 35% lower RMSE under controlled lighting shifts.
69b90f3a6b3f36c249fbc609
Haoze Zheng
admin_assigned
2026-03-17T08:46:43.854Z
662a8cc138c02c106f883183
https://cdn-avatars.huggingface.co/v1/production/uploads/662a8cc138c02c106f883183/FvWqqawwAPA-aEGoPt6i5.jpeg
Haoze Zheng
Zanghu-Ze
user
Zanghu-Ze
69b90f3a6b3f36c249fbc60a
Zihao Wang
69b90f3a6b3f36c249fbc60b
Xianfeng Wu
69b90f3a6b3f36c249fbc60c
Yajing Bai
admin_assigned
2026-03-17T08:46:57.887Z
66e15e6a9cbd604971999bf2
/avatars/dc02874bbe33e33c9f0dad4b40e55e01.svg
Yajing Bai
YajingB
user
YajingB
69b90f3a6b3f36c249fbc60d
Yexin Liu
admin_assigned
2026-03-17T08:47:04.715Z
6497ff395b5d43c1c77d2a11
/avatars/e3704701ed3ea0c1c165001521f40086.svg
Yexin Liu
AIPeanutman
user
AIPeanutman
69b90f3a6b3f36c249fbc60e
Yun Li
69b90f3a6b3f36c249fbc60f
Xiaogang Xu
admin_assigned
2026-03-17T08:47:17.806Z
637ee45b2438d7485b8d8f6a
/avatars/11b7d29b6fa6c1b392641e0cd4002863.svg
Xiaogang Xu
xiaogang00
user
xiaogang00
69b90f3a6b3f36c249fbc610
Harry Yang
Learning Latent Proxies for Controllable Single-Image Relighting
Single-image relighting is highly under-constrained: small illumination changes can produce large, nonlinear variations in shading, shadows, and specularities, while geometry and materials remain unobserved. Existing diffusion-based approaches either rely on intrinsic or G-buffer pipelines that require dense and fragile supervision, or operate purely in latent space without physical grounding, making fine-grained control of direction, intensity, and color unreliable. We observe that a full intrinsic decomposition is unnecessary and redundant for accurate relighting. Instead, sparse but physically meaningful cues, indicating where illumination should change and how materials should respond, are sufficient to guide a diffusion model. Based on this insight, we introduce LightCtrl that integrates physical priors at two levels: a few-shot latent proxy encoder that extracts compact material-geometry cues from limited PBR supervision, and a lighting-aware mask that identifies sensitive illumination regions and steers the denoiser toward shading relevant pixels. To compensate for scarce PBR data, we refine the proxy branch using a DPO-based objective that enforces physical consistency in the predicted cues. We also present ScaLight, a large-scale object-level dataset with systematically varied illumination and complete camera-light metadata, enabling physically consistent and controllable training. Across object and scene level benchmarks, our method achieves photometrically faithful relighting with accurate continuous control, surpassing prior diffusion and intrinsic-based baselines, including gains of up to +2.4 dB PSNR and 35% lower RMSE under controlled lighting shifts.