Rethinking Visual Intelligence: Insights from Video Pretraining
Large language models (LLMs) have demonstrated that large-scale pretraining
enables systems to adapt rapidly to new problems with little supervision in the
language domain. This success, however, has not translated as effectively to
the visual domain, where models, including LLMs, continue to struggle with
compositional understanding, sample efficiency, and general-purpose
problem-solving. We investigate Video Diffusion Models (VDMs) as a promising
direction for bridging this gap. Pretraining on spatiotemporal data endows
these models with strong inductive biases for structure and dynamics, which we
hypothesize can support broad task adaptability. To test this, we design a
controlled evaluation in which both a pretrained LLM and a pretrained VDM are
equipped with lightweight adapters and presented with tasks in their natural
modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games,
route planning, and cellular automata, VDMs demonstrate higher data efficiency
than their language counterparts. Taken together, our results indicate that
video pretraining offers inductive biases that support progress toward visual
foundation models.
6901734b646208eac0d1f363
Pablo Acuaviva
6901734b646208eac0d1f364
Aram Davtyan
claimed_verified
2026-02-11T11:21:46.114Z
655df11f82afda0fc47c421d
/avatars/f0ea4317e6010b28392d85bb94dd2230.svg
Aram Davtyan
araachie
user
araachie
6901734b646208eac0d1f365
Mariam Hassan
6901734b646208eac0d1f366
Sebastian Stapf
6901734b646208eac0d1f367
Ahmad Rahimi
6901734b646208eac0d1f368
Alexandre Alahi
6901734b646208eac0d1f369
Paolo Favaro
Rethinking Visual Intelligence: Insights from Video Pretraining
Large language models (LLMs) have demonstrated that large-scale pretraining
enables systems to adapt rapidly to new problems with little supervision in the
language domain. This success, however, has not translated as effectively to
the visual domain, where models, including LLMs, continue to struggle with
compositional understanding, sample efficiency, and general-purpose
problem-solving. We investigate Video Diffusion Models (VDMs) as a promising
direction for bridging this gap. Pretraining on spatiotemporal data endows
these models with strong inductive biases for structure and dynamics, which we
hypothesize can support broad task adaptability. To test this, we design a
controlled evaluation in which both a pretrained LLM and a pretrained VDM are
equipped with lightweight adapters and presented with tasks in their natural
modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games,
route planning, and cellular automata, VDMs demonstrate higher data efficiency
than their language counterparts. Taken together, our results indicate that
video pretraining offers inductive biases that support progress toward visual
foundation models.
Video Diffusion Models (VDMs) show higher data efficiency than Large Language Models (LLMs) across various visual tasks, suggesting video pretraining can enhance visual foundation models.
Video Diffusion Models VDMs Large Language Models LLMs spatiotemporal data inductive biases ARC-AGI ConceptARC visual games route planning cellular automata data efficiency visual foundation models