4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation
Despite advances in Multimodal LLMs (MLLMs), their ability to reason over 3D structures and temporal dynamics remains limited, constrained by weak 4D perception and temporal understanding. Existing 3D and 4D Video Question Answering (VQA) benchmarks also emphasize static scenes and lack region-level prompting. We tackle these issues by introducing: (a) 4D-RGPT, a specialized MLLM designed to capture 4D representations from video inputs with enhanced temporal perception; (b) Perceptual 4D Distillation (P4D), a training framework that transfers 4D representations from a frozen expert model into 4D-RGPT for comprehensive 4D perception; and (c) R4D-Bench, a benchmark for depth-aware dynamic scenes with region-level prompting, built via a hybrid automated and human-verified pipeline. Our 4D-RGPT achieves notable improvements on both existing 4D VQA benchmarks and the proposed R4D-Bench benchmark.
6948aa3d34f46eaf46cbb1cb
Chiao-An Yang
6948aa3d34f46eaf46cbb1cc
Ryo Hachiuma
6948aa3d34f46eaf46cbb1cd
Sifei Liu
6948aa3d34f46eaf46cbb1ce
Subhashree Radhakrishnan
6948aa3d34f46eaf46cbb1cf
Raymond A. Yeh
6948aa3d34f46eaf46cbb1d0
Yu-Chiang Frank Wang
6948aa3d34f46eaf46cbb1d1
Min-Hung Chen
claimed_verified
2025-12-22T10:58:54.795Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
cmhungsteve
user
cmhungsteve
4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation
Despite advances in Multimodal LLMs (MLLMs), their ability to reason over 3D structures and temporal dynamics remains limited, constrained by weak 4D perception and temporal understanding. Existing 3D and 4D Video Question Answering (VQA) benchmarks also emphasize static scenes and lack region-level prompting. We tackle these issues by introducing: (a) 4D-RGPT, a specialized MLLM designed to capture 4D representations from video inputs with enhanced temporal perception; (b) Perceptual 4D Distillation (P4D), a training framework that transfers 4D representations from a frozen expert model into 4D-RGPT for comprehensive 4D perception; and (c) R4D-Bench, a benchmark for depth-aware dynamic scenes with region-level prompting, built via a hybrid automated and human-verified pipeline. Our 4D-RGPT achieves notable improvements on both existing 4D VQA benchmarks and the proposed R4D-Bench benchmark.
4D-RGPT, a specialized multimodal LLM, enhances 4D perception in video inputs through Perceptual 4D Distillation and is evaluated on R4D-Bench, a new benchmark for depth-aware dynamic scenes.
4D-RGPT Multimodal LLMs 4D Video Question Answering 4D representations Perceptual 4D Distillation R4D-Bench depth-aware dynamic scenes region-level prompting