Future Optical Flow Prediction Improves Robot Control & Video Generation
Future motion representations, such as optical flow, offer immense value for control and generative tasks. However, forecasting generalizable spatially dense motion representations remains a key challenge, and learning such forecasting from noisy, real-world data remains relatively unexplored. We introduce FOFPred, a novel language-conditioned optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture. This unique combination enables strong multimodal reasoning with pixel-level generative fidelity for future motion prediction. Our model is trained on web-scale human activity data-a highly scalable but unstructured source. To extract meaningful signals from this noisy video-caption data, we employ crucial data preprocessing techniques and our unified architecture with strong image pretraining. The resulting trained model is then extended to tackle two distinct downstream tasks in control and generation. Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred, confirming the value of a unified VLM-Diffusion architecture and scalable learning from diverse web data for future optical flow prediction.
696d9f043f1837bfb89709b6
Kanchana Ranasinghe
696d9f043f1837bfb89709b7
Honglu Zhou
696d9f043f1837bfb89709b8
Yu Fang
696d9f043f1837bfb89709b9
Luyu Yang
696d9f043f1837bfb89709ba
Le Xue
696d9f043f1837bfb89709bb
Ran Xu
696d9f043f1837bfb89709bc
Caiming Xiong
696d9f043f1837bfb89709bd
Silvio Savarese
696d9f043f1837bfb89709be
Michael S Ryoo
696d9f043f1837bfb89709bf
Juan Carlos Niebles
Future Optical Flow Prediction Improves Robot Control & Video Generation
Future motion representations, such as optical flow, offer immense value for control and generative tasks. However, forecasting generalizable spatially dense motion representations remains a key challenge, and learning such forecasting from noisy, real-world data remains relatively unexplored. We introduce FOFPred, a novel language-conditioned optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture. This unique combination enables strong multimodal reasoning with pixel-level generative fidelity for future motion prediction. Our model is trained on web-scale human activity data-a highly scalable but unstructured source. To extract meaningful signals from this noisy video-caption data, we employ crucial data preprocessing techniques and our unified architecture with strong image pretraining. The resulting trained model is then extended to tackle two distinct downstream tasks in control and generation. Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred, confirming the value of a unified VLM-Diffusion architecture and scalable learning from diverse web data for future optical flow prediction.
A novel language-conditioned optical flow forecasting model combines Vision-Language Model and Diffusion architecture to predict future motion from noisy web-scale video data, demonstrating versatility in robotic manipulation and video generation tasks.
Vision-Language Model Diffusion architecture optical flow forecasting multimodal reasoning pixel-level generative fidelity web-scale human activity data data preprocessing image pretraining robotic manipulation video generation