SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead
Vision-Language-Action (VLA) models built on pretrained Vision-Language Models (VLMs) show strong potential but are limited in practicality due to their large parameter counts. To mitigate this issue, using a lightweight VLM has been explored, but it compromises spatiotemporal reasoning. Although some methods suggest that incorporating additional 3D inputs can help, they usually rely on large VLMs to fuse 3D and 2D inputs and still lack temporal understanding. Therefore, we propose SwiftVLA, an architecture that enhances a compact model with 4D understanding while preserving design efficiency. Specifically, our approach features a pretrained 4D visual geometry transformer with a temporal cache that extracts 4D features from 2D images. Then, to enhance the VLM's ability to exploit both 2D images and 4D features, we introduce Fusion Tokens, a set of learnable tokens trained with a future prediction objective to generate unified representations for action generation. Finally, we introduce a mask-and-reconstruct strategy that masks 4D inputs to the VLM and trains the VLA to reconstruct them, enabling the VLM to learn effective 4D representations and allowing the 4D branch to be dropped at inference with minimal performance loss. Experiments in real and simulated environments show that SwiftVLA outperforms lightweight baselines and rivals VLAs up to 7 times larger, achieving comparable performance on edge devices while being 18 times faster and reducing memory footprint by 12 times.
692fbdef26742347f61dacaf
Chaojun Ni
692fbdef26742347f61dacb0
Cheng Chen
692fbdef26742347f61dacb1
Xiaofeng Wang
claimed_verified
2025-12-03T09:15:30.398Z
6426616ea5ec4a5cbc535634
https://cdn-avatars.huggingface.co/v1/production/uploads/6426616ea5ec4a5cbc535634/5IfSFYd9QOxz8K9QmBCst.png
JeffWang
Jeff-Wang
user
Jeff-Wang
692fbdef26742347f61dacb2
Zheng Zhu
claimed_verified
2025-12-03T09:15:32.398Z
656e9b562cd7a3e348011d26
/avatars/bcca51bdc27c664f8f132420e6ed99fa.svg
Zheng Zhu
ZhengZhu
user
ZhengZhu
692fbdef26742347f61dacb3
Wenzhao Zheng
692fbdef26742347f61dacb4
Boyuan Wang
692fbdef26742347f61dacb5
Tianrun Chen
692fbdef26742347f61dacb6
Guosheng Zhao
692fbdef26742347f61dacb7
Haoyun Li
692fbdef26742347f61dacb8
Zhehao Dong
692fbdef26742347f61dacb9
Qiang Zhang
692fbdef26742347f61dacba
Yun Ye
692fbdef26742347f61dacbb
Yang Wang
692fbdef26742347f61dacbc
Guan Huang
692fbdef26742347f61dacbd
Wenjun Mei
SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead
Vision-Language-Action (VLA) models built on pretrained Vision-Language Models (VLMs) show strong potential but are limited in practicality due to their large parameter counts. To mitigate this issue, using a lightweight VLM has been explored, but it compromises spatiotemporal reasoning. Although some methods suggest that incorporating additional 3D inputs can help, they usually rely on large VLMs to fuse 3D and 2D inputs and still lack temporal understanding. Therefore, we propose SwiftVLA, an architecture that enhances a compact model with 4D understanding while preserving design efficiency. Specifically, our approach features a pretrained 4D visual geometry transformer with a temporal cache that extracts 4D features from 2D images. Then, to enhance the VLM's ability to exploit both 2D images and 4D features, we introduce Fusion Tokens, a set of learnable tokens trained with a future prediction objective to generate unified representations for action generation. Finally, we introduce a mask-and-reconstruct strategy that masks 4D inputs to the VLM and trains the VLA to reconstruct them, enabling the VLM to learn effective 4D representations and allowing the 4D branch to be dropped at inference with minimal performance loss. Experiments in real and simulated environments show that SwiftVLA outperforms lightweight baselines and rivals VLAs up to 7 times larger, achieving comparable performance on edge devices while being 18 times faster and reducing memory footprint by 12 times.
SwiftVLA enhances compact Vision-Language-Action models with 4D understanding using Fusion Tokens and a mask-and-reconstruct strategy, achieving high performance with reduced computational and memory demands.
pretrained Vision-Language Models VLA models spatiotemporal reasoning lightweight VLM 3D inputs 4D visual geometry transformer temporal cache Fusion Tokens future prediction objective mask-and-reconstruct strategy 4D representations edge devices