Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes
We introduce Talk2Move, a reinforcement learning (RL) based diffusion framework for text-instructed spatial transformation of objects within scenes. Spatially manipulating objects in a scene through natural language poses a challenge for multimodal generation systems. While existing text-based manipulation methods can adjust appearance or style, they struggle to perform object-level geometric transformations-such as translating, rotating, or resizing objects-due to scarce paired supervision and pixel-level optimization limits. Talk2Move employs Group Relative Policy Optimization (GRPO) to explore geometric actions through diverse rollouts generated from input images and lightweight textual variations, removing the need for costly paired data. A spatial reward guided model aligns geometric transformations with linguistic description, while off-policy step evaluation and active step sampling improve learning efficiency by focusing on informative transformation stages. Furthermore, we design object-centric spatial rewards that evaluate displacement, rotation, and scaling behaviors directly, enabling interpretable and coherent transformations. Experiments on curated benchmarks demonstrate that Talk2Move achieves precise, consistent, and semantically faithful object transformations, outperforming existing text-guided editing approaches in both spatial accuracy and scene coherence.
695c813c6aa73bc11f091475
Jing Tan
claimed_verified
2026-01-06T09:57:49.343Z
65367c40061949598892dbdc
/avatars/4baf27263841471cbd5f629a8b99424d.svg
Jing Tan
jingtan
user
jingtan
695c813c6aa73bc11f091476
Zhaoyang Zhang
claimed_verified
2026-01-06T09:57:47.322Z
658409ceca19ccf6d9989add
/avatars/3ac1dbd25e52435185babdeb3da28875.svg
Zhaoyang Zhang
ZyZcuhk
user
ZyZcuhk
695c813c6aa73bc11f091477
Yantao Shen
695c813c6aa73bc11f091478
Jiarui Cai
695c813c6aa73bc11f091479
Shuo Yang
695c813c6aa73bc11f09147a
Jiajun Wu
695c813c6aa73bc11f09147b
Wei Xia
695c813c6aa73bc11f09147c
Zhuowen Tu
695c813c6aa73bc11f09147d
Stefano Soatto
Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes
We introduce Talk2Move, a reinforcement learning (RL) based diffusion framework for text-instructed spatial transformation of objects within scenes. Spatially manipulating objects in a scene through natural language poses a challenge for multimodal generation systems. While existing text-based manipulation methods can adjust appearance or style, they struggle to perform object-level geometric transformations-such as translating, rotating, or resizing objects-due to scarce paired supervision and pixel-level optimization limits. Talk2Move employs Group Relative Policy Optimization (GRPO) to explore geometric actions through diverse rollouts generated from input images and lightweight textual variations, removing the need for costly paired data. A spatial reward guided model aligns geometric transformations with linguistic description, while off-policy step evaluation and active step sampling improve learning efficiency by focusing on informative transformation stages. Furthermore, we design object-centric spatial rewards that evaluate displacement, rotation, and scaling behaviors directly, enabling interpretable and coherent transformations. Experiments on curated benchmarks demonstrate that Talk2Move achieves precise, consistent, and semantically faithful object transformations, outperforming existing text-guided editing approaches in both spatial accuracy and scene coherence.
Talk2Move presents a reinforcement learning-based diffusion framework that enables precise, semantically faithful spatial transformations of objects in scenes using natural language instructions.
reinforcement learning diffusion framework text-instructed spatial transformation Group Relative Policy Optimization geometric actions spatial reward guided model off-policy step evaluation active step sampling object-centric spatial rewards spatial accuracy scene coherence