ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination
Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/
69c14864e9374d7b3df94d26
Jan-Niklas Dihlmann
claimed_verified
2026-03-24T09:22:53.777Z
631b6dbcbf1351ed2bd05be1
https://cdn-avatars.huggingface.co/v1/production/uploads/631b6dbcbf1351ed2bd05be1/_kzcDD4qCifsb-651_LmY.png
Jan-Niklas Dihlmann
JDihlmann
user
JDihlmann
69c14864e9374d7b3df94d27
Mark Boss
69c14864e9374d7b3df94d28
Simon Donne
69c14864e9374d7b3df94d29
Andreas Engelhardt
69c14864e9374d7b3df94d2a
Hendrik P. A. Lensch
69c14864e9374d7b3df94d2b
Varun Jampani
ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination
Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/
ReLi3D presents a unified end-to-end pipeline for simultaneous 3D geometry, material, and illumination reconstruction from multi-view images using transformer cross-conditioning and a two-path prediction strategy.
multi-view constraints transformer cross-conditioning architecture unified two-path prediction strategy differentiable Monte Carlo multiple importance sampling renderer mixed domain training protocol synthetic PBR datasets real-world RGB captures