Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning
Reinforcement learning has empowered large language models to act as intelligent agents, yet training them for long-horizon tasks remains challenging due to the scarcity of high-quality trajectories, especially under limited resources. Existing methods typically scale up rollout sizes and indiscriminately allocate computational resources among intermediate steps. Such attempts inherently waste substantial computation budget on trivial steps while failing to guarantee sample quality. To address this, we propose Spark (Strategic Policy-Aware exploRation via Key-state dynamic branching), a novel framework that selectively branches at critical decision states for resource-efficient exploration. Our key insight is to activate adaptive branching exploration at critical decision points to probe promising trajectories, thereby achieving precise resource allocation that prioritizes sampling quality over blind coverage. This design leverages the agent's intrinsic decision-making signals to reduce dependence on human priors, enabling the agent to autonomously expand exploration and achieve stronger generalization. Experiments across diverse tasks (e.g., embodied planning), demonstrate that Spark achieves superior success rates with significantly fewer training samples, exhibiting robust generalization even in unseen scenarios.
697acf8ddf3e800774f13c46
Jinyang Wu
claimed_verified
2026-01-29T09:16:13.804Z
6747de57f8cab58c22ec94a2
/avatars/5bae0341862fac24564781c0fa32aac5.svg
Jinyang Wu
Jinyang23
user
Jinyang23
697acf8ddf3e800774f13c47
Shuo Yang
697acf8ddf3e800774f13c48
Changpeng Yang
697acf8ddf3e800774f13c49
Yuhao Shen
697acf8ddf3e800774f13c4a
Shuai Zhang
697acf8ddf3e800774f13c4b
Zhengqi Wen
697acf8ddf3e800774f13c4c
Jianhua Tao
Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning
Reinforcement learning has empowered large language models to act as intelligent agents, yet training them for long-horizon tasks remains challenging due to the scarcity of high-quality trajectories, especially under limited resources. Existing methods typically scale up rollout sizes and indiscriminately allocate computational resources among intermediate steps. Such attempts inherently waste substantial computation budget on trivial steps while failing to guarantee sample quality. To address this, we propose Spark (Strategic Policy-Aware exploRation via Key-state dynamic branching), a novel framework that selectively branches at critical decision states for resource-efficient exploration. Our key insight is to activate adaptive branching exploration at critical decision points to probe promising trajectories, thereby achieving precise resource allocation that prioritizes sampling quality over blind coverage. This design leverages the agent's intrinsic decision-making signals to reduce dependence on human priors, enabling the agent to autonomously expand exploration and achieve stronger generalization. Experiments across diverse tasks (e.g., embodied planning), demonstrate that Spark achieves superior success rates with significantly fewer training samples, exhibiting robust generalization even in unseen scenarios.
Spark is a reinforcement learning framework that strategically allocates computational resources by branching at critical decision states, improving sample efficiency and generalization for long-horizon tasks.
reinforcement learning large language models long-horizon tasks trajectory scarcity rollout size computational resource allocation adaptive branching exploration decision-making signals sample efficiency generalization