VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs
Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90 percent on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.
69c3733395b48f16a3007883
Haoran Yuan
69c3733395b48f16a3007884
Weigang Yi
69c3733395b48f16a3007885
Zhenyu Zhang
69c3733395b48f16a3007886
Wendi Chen
69c3733395b48f16a3007887
Yuchen Mo
69c3733395b48f16a3007888
Jiashi Yin
69c3733395b48f16a3007889
Xinzhuo Li
69c3733395b48f16a300788a
Xiangyu Zeng
69c3733395b48f16a300788b
Chuan Wen
69c3733395b48f16a300788c
Cewu Lu
69c3733395b48f16a300788d
Katherine Driggs-Campbell
69c3733395b48f16a300788e
Ismini Lourentzou
claimed_verified
2026-04-13T07:37:37.393Z
678ac3b31cbaa0b4bc295885
/avatars/1244fc1b305c9c6383df9bb5e4707347.svg
Ismini Lourentzou
isminoula
user
isminoula
VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs
Video-Action Models (VAMs) have emerged as a promising framework for embodied intelligence, learning implicit world dynamics from raw video streams to produce temporally consistent action predictions. Although such models demonstrate strong performance on long-horizon tasks through visual reasoning, they remain limited in contact-rich scenarios where critical interaction states are only partially observable from vision alone. In particular, fine-grained force modulation and contact transitions are not reliably encoded in visual tokens, leading to unstable or imprecise behaviors. To bridge this gap, we introduce the Video-Tactile Action Model (VTAM), a multimodal world modeling framework that incorporates tactile perception as a complementary grounding signal. VTAM augments a pretrained video transformer with tactile streams via a lightweight modality transfer finetuning, enabling efficient cross-modal representation learning without tactile-language paired data or independent tactile pretraining. To stabilize multimodal fusion, we introduce a tactile regularization loss that enforces balanced cross-modal attention, preventing visual latent dominance in the action model. VTAM demonstrates superior performance in contact-rich manipulation, maintaining a robust success rate of 90 percent on average. In challenging scenarios such as potato chip pick-and-place requiring high-fidelity force awareness, VTAM outperforms the pi 0.5 baseline by 80 percent. Our findings demonstrate that integrating tactile feedback is essential for correcting visual estimation errors in world action models, providing a scalable approach to physically grounded embodied foundation models.
Video-Tactile Action Models combine visual and tactile perception to improve contact-rich manipulation tasks through multimodal fusion and regularization techniques.
Video-Action Models embodied intelligence visual reasoning contact-rich scenarios visual tokens force modulation contact transitions multimodal world modeling tactile perception video transformer modality transfer finetuning cross-modal representation learning tactile regularization loss cross-modal attention action model embodied foundation models