FrameDiffuser: G-Buffer-Conditioned Diffusion for Neural Forward Frame Rendering
Neural rendering for interactive applications requires translating geometric and material properties (G-buffer) to photorealistic images with realistic lighting on a frame-by-frame basis. While recent diffusion-based approaches show promise for G-buffer-conditioned image synthesis, they face critical limitations: single-image models like RGBX generate frames independently without temporal consistency, while video models like DiffusionRenderer are too computationally expensive for most consumer gaming sets ups and require complete sequences upfront, making them unsuitable for interactive applications where future frames depend on user input. We introduce FrameDiffuser, an autoregressive neural rendering framework that generates temporally consistent, photorealistic frames by conditioning on G-buffer data and the models own previous output. After an initial frame, FrameDiffuser operates purely on incoming G-buffer data, comprising geometry, materials, and surface properties, while using its previously generated frame for temporal guidance, maintaining stable, temporal consistent generation over hundreds to thousands of frames. Our dual-conditioning architecture combines ControlNet for structural guidance with ControlLoRA for temporal coherence. A three-stage training strategy enables stable autoregressive generation. We specialize our model to individual environments, prioritizing consistency and inference speed over broad generalization, demonstrating that environment-specific training achieves superior photorealistic quality with accurate lighting, shadows, and reflections compared to generalized approaches.
694526cbfbf17e708e186298
Ole Beisswenger
claimed_verified
2025-12-25T20:52:22.949Z
675af2c733df760d74e8b0ae
/avatars/6b61b69c6521adc81bc963cca2742d09.svg
Ole Beißwenger
Obeisswenger
user
Obeisswenger
694526cbfbf17e708e186299
Jan-Niklas Dihlmann
claimed_verified
2025-12-19T10:49:54.174Z
631b6dbcbf1351ed2bd05be1
https://cdn-avatars.huggingface.co/v1/production/uploads/631b6dbcbf1351ed2bd05be1/_kzcDD4qCifsb-651_LmY.png
Jan-Niklas Dihlmann
JDihlmann
user
JDihlmann
694526cbfbf17e708e18629a
Hendrik P. A. Lensch
FrameDiffuser: G-Buffer-Conditioned Diffusion for Neural Forward Frame Rendering
Neural rendering for interactive applications requires translating geometric and material properties (G-buffer) to photorealistic images with realistic lighting on a frame-by-frame basis. While recent diffusion-based approaches show promise for G-buffer-conditioned image synthesis, they face critical limitations: single-image models like RGBX generate frames independently without temporal consistency, while video models like DiffusionRenderer are too computationally expensive for most consumer gaming sets ups and require complete sequences upfront, making them unsuitable for interactive applications where future frames depend on user input. We introduce FrameDiffuser, an autoregressive neural rendering framework that generates temporally consistent, photorealistic frames by conditioning on G-buffer data and the models own previous output. After an initial frame, FrameDiffuser operates purely on incoming G-buffer data, comprising geometry, materials, and surface properties, while using its previously generated frame for temporal guidance, maintaining stable, temporal consistent generation over hundreds to thousands of frames. Our dual-conditioning architecture combines ControlNet for structural guidance with ControlLoRA for temporal coherence. A three-stage training strategy enables stable autoregressive generation. We specialize our model to individual environments, prioritizing consistency and inference speed over broad generalization, demonstrating that environment-specific training achieves superior photorealistic quality with accurate lighting, shadows, and reflections compared to generalized approaches.
FrameDiffuser is an autoregressive neural rendering framework that generates temporally consistent photorealistic frames using G-buffer data and previous frame outputs, leveraging ControlNet and ControlLoRA for structural and temporal coherence.
neural rendering G-buffer diffusion-based approaches RGBX DiffusionRenderer autoregressive framework temporal consistency ControlNet ControlLoRA three-stage training structural guidance temporal coherence environment-specific training