ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.
69c33dbc95b48f16a3007714
Haichao Zhang
claimed_verified
2026-04-17T06:59:28.245Z
66393f5a1231260674ae798e
https://cdn-avatars.huggingface.co/v1/production/uploads/66393f5a1231260674ae798e/JlWLdLTjMwgh0x1ASx4Mx.jpeg
Haichao Zhang
haichaozhang
user
haichaozhang
69c33dbc95b48f16a3007715
Yijiang Li
69c33dbc95b48f16a3007716
Shwai He
claimed_verified
2026-04-13T07:37:39.436Z
64468d7a3b28db22e7f84418
/avatars/ff05994acd7da64a2ed6e44f12cf77bc.svg
Shwai He
shwai-he
user
shwai-he
69c33dbc95b48f16a3007717
Tushar Nagarajan
69c33dbc95b48f16a3007718
Mingfei Chen
69c33dbc95b48f16a3007719
Jianglin Lu
69c33dbc95b48f16a300771a
Ang Li
69c33dbc95b48f16a300771b
Yun Fu
ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.
A VLM-guided JEPA-style latent world modeling framework combines dense-frame dynamics modeling with long-horizon semantic guidance through dual-temporal pathways to improve hand-manipulation trajectory prediction.
latent world models V-JEPA2 vision-language models JEPA-style latent world modeling dense-frame dynamics modeling long-horizon semantic guidance dual-temporal pathway VLM thinker branch hierarchical pyramid representation extraction module hand-manipulation trajectory prediction