Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.
69b7b5b691a28162b233fd02
Yao Zhang
69b7b5b691a28162b233fd03
Zhuchenyang Liu
claimed_verified
2026-03-16T10:04:42.595Z
66425e128a30a66d99c18c1f
https://cdn-avatars.huggingface.co/v1/production/uploads/66425e128a30a66d99c18c1f/LzNnPHthroClxSYb_Ix0j.jpeg
Zhuchenyang Liu
Ryenhails
user
Ryenhails
69b7b5b691a28162b233fd04
Yanlan He
69b7b5b691a28162b233fd05
Thomas Ploetz
69b7b5b691a28162b233fd06
Yu Xiao
Fine-grained Motion Retrieval via Joint-Angle Motion Images and Token-Patch Late Interaction
Text-motion retrieval aims to learn a semantically aligned latent space between natural language descriptions and 3D human motion skeleton sequences, enabling bidirectional search across the two modalities. Most existing methods use a dual-encoder framework that compresses motion and text into global embeddings, discarding fine-grained local correspondences, and thus reducing accuracy. Additionally, these global-embedding methods offer limited interpretability of the retrieval results. To overcome these limitations, we propose an interpretable, joint-angle-based motion representation that maps joint-level local features into a structured pseudo-image, compatible with pre-trained Vision Transformers. For text-to-motion retrieval, we employ MaxSim, a token-wise late interaction mechanism, and enhance it with Masked Language Modeling regularization to foster robust, interpretable text-motion alignment. Extensive experiments on HumanML3D and KIT-ML show that our method outperforms state-of-the-art text-motion retrieval approaches while offering interpretable fine-grained correspondences between text and motion. The code is available in the supplementary material.
A novel text-motion retrieval approach uses joint-angle-based motion representation and Vision Transformer-compatible pseudo-images to achieve superior performance with interpretable fine-grained alignments.
dual-encoder framework global embeddings Vision Transformers MaxSim token-wise late interaction Masked Language Modeling joint-level local features structured pseudo-image