Act2Goal: From World Model To General Goal-conditioned Policy
Specifying robotic manipulation tasks in a manner that is both expressive and precise remains a central challenge. While visual goals provide a compact and unambiguous task specification, existing goal-conditioned policies often struggle with long-horizon manipulation due to their reliance on single-step action prediction without explicit modeling of task progress. We propose Act2Goal, a general goal-conditioned manipulation policy that integrates a goal-conditioned visual world model with multi-scale temporal control. Given a current observation and a target visual goal, the world model generates a plausible sequence of intermediate visual states that captures long-horizon structure. To translate this visual plan into robust execution, we introduce Multi-Scale Temporal Hashing (MSTH), which decomposes the imagined trajectory into dense proximal frames for fine-grained closed-loop control and sparse distal frames that anchor global task consistency. The policy couples these representations with motor control through end-to-end cross-attention, enabling coherent long-horizon behavior while remaining reactive to local disturbances. Act2Goal achieves strong zero-shot generalization to novel objects, spatial layouts, and environments. We further enable reward-free online adaptation through hindsight goal relabeling with LoRA-based finetuning, allowing rapid autonomous improvement without external supervision. Real-robot experiments demonstrate that Act2Goal improves success rates from 30% to 90% on challenging out-of-distribution tasks within minutes of autonomous interaction, validating that goal-conditioned world models with multi-scale temporal control provide structured guidance necessary for robust long-horizon manipulation. Project page: https://act2goal.github.io/
69534cef89916ff627aa3f77
Pengfei Zhou
69534cef89916ff627aa3f78
Liliang Chen
claimed_verified
2025-12-31T20:56:46.231Z
66c4816e96583c59b09fec30
https://cdn-avatars.huggingface.co/v1/production/uploads/66c4816e96583c59b09fec30/RLurCsmcgOfyuWVkpsv1L.jpeg
Ryan Chen
ryancll118
user
ryancll118
69534cef89916ff627aa3f79
Shengcong Chen
69534cef89916ff627aa3f7a
Di Chen
69534cef89916ff627aa3f7b
Wenzhi Zhao
69534cef89916ff627aa3f7c
Rongjun Jin
69534cef89916ff627aa3f7d
Guanghui Ren
claimed_verified
2025-12-31T20:56:44.283Z
646ec9b135f55eb49e405faa
/avatars/a17194be585d20e2a021e77a5a20e213.svg
Guanghui Ren
sundrops
user
sundrops
69534cef89916ff627aa3f7e
Jianlan Luo
Act2Goal: From World Model To General Goal-conditioned Policy
Specifying robotic manipulation tasks in a manner that is both expressive and precise remains a central challenge. While visual goals provide a compact and unambiguous task specification, existing goal-conditioned policies often struggle with long-horizon manipulation due to their reliance on single-step action prediction without explicit modeling of task progress. We propose Act2Goal, a general goal-conditioned manipulation policy that integrates a goal-conditioned visual world model with multi-scale temporal control. Given a current observation and a target visual goal, the world model generates a plausible sequence of intermediate visual states that captures long-horizon structure. To translate this visual plan into robust execution, we introduce Multi-Scale Temporal Hashing (MSTH), which decomposes the imagined trajectory into dense proximal frames for fine-grained closed-loop control and sparse distal frames that anchor global task consistency. The policy couples these representations with motor control through end-to-end cross-attention, enabling coherent long-horizon behavior while remaining reactive to local disturbances. Act2Goal achieves strong zero-shot generalization to novel objects, spatial layouts, and environments. We further enable reward-free online adaptation through hindsight goal relabeling with LoRA-based finetuning, allowing rapid autonomous improvement without external supervision. Real-robot experiments demonstrate that Act2Goal improves success rates from 30% to 90% on challenging out-of-distribution tasks within minutes of autonomous interaction, validating that goal-conditioned world models with multi-scale temporal control provide structured guidance necessary for robust long-horizon manipulation. Project page: https://act2goal.github.io/
Act2Goal employs a goal-conditioned visual world model with multi-scale temporal control and cross-attention to achieve robust long-horizon robotic manipulation through structured planning and adaptive execution.
goal-conditioned visual world model Multi-Scale Temporal Hashing (MSTH) cross-attention end-to-end cross-attention LoRA-based finetuning hindsight goal relabeling