OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams
Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.
69b372e32ad6c5d4cf4869d5
Yibin Yan
admin_assigned
2026-03-13T09:03:48.398Z
62dab78c4be126e22e8ed1c8
/avatars/72e81067aa942f21226876844836c6e3.svg
Yibin Yan(SII)
Go2Heart
user
Go2Heart
69b372e32ad6c5d4cf4869d6
Jilan Xu
admin_assigned
2026-03-13T09:03:57.605Z
642e20d8018d9a57ca52af85
/avatars/c99c875a6654e7a8b3958777bcc46d8f.svg
Jilan Xu
Jazzcharles
user
Jazzcharles
69b372e32ad6c5d4cf4869d7
Shangzhe Di
69b372e32ad6c5d4cf4869d8
Haoning Wu
claimed_verified
2026-03-13T08:13:00.437Z
632c7a0d1d303f5f9acf01b8
https://cdn-avatars.huggingface.co/v1/production/uploads/632c7a0d1d303f5f9acf01b8/T010IFuCp6UaOeIyWhbCk.jpeg
Haoning Wu
haoningwu
user
haoningwu
69b372e32ad6c5d4cf4869d9
Weidi Xie
OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams
Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.
OmniStream is a unified visual backbone that processes streaming video data through causal spatiotemporal attention and 3D rotary positional embeddings, enabling general-purpose visual understanding across multiple domains.
causal spatiotemporal attention 3D rotary positional embeddings persistent KV-cache multi-task framework vision-language alignment streaming geometric reconstruction video probing spatial reasoning temporal reasoning robotic manipulation