ProAct: Agentic Lookahead in Interactive Environments
Existing Large Language Model (LLM) agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm. First, we introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search. By compressing complex search trees into concise, causal reasoning chains, the agent learns the logic of foresight without the computational overhead of inference-time search. Second, to further refine decision accuracy, we propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms like PPO and GRPO. By leveraging lightweight environment rollouts to calibrate value estimates, MC-Critic provides a low-variance signal that facilitates stable policy optimization without relying on expensive model-based value approximation. Experiments on both stochastic (e.g., 2048) and deterministic (e.g., Sokoban) environments demonstrate that ProAct significantly improves planning accuracy. Notably, a 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while demonstrating robust generalization to unseen environments. The codes and models are available at https://github.com/GreatX3/ProAct
698557724ad556f294b7eafb
Yangbin Yu
698557724ad556f294b7eafc
Mingyu Yang
698557724ad556f294b7eafd
Junyou Li
698557724ad556f294b7eafe
Yiming Gao
claimed_verified
2026-02-06T18:52:08.424Z
67cff2a43a9d50150f6d55e6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/iY05tc7uuocF55aT08Vdx.png
Yatming Gor
code-yatming
user
code-yatming
698557724ad556f294b7eaff
Feiyu Liu
698557724ad556f294b7eb00
Yijun Yang
claimed_verified
2026-03-26T14:04:51.810Z
645e553c3b6d85c65e8b0e54
/avatars/1fffc6499b9d65b21a895ca96f03b781.svg
Steven
yijunyang
user
yijunyang
698557724ad556f294b7eb01
Zichuan Lin
claimed_verified
2026-03-26T14:04:48.016Z
687da36e2eaea8261f1323d6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hFH69bJGIDMikEYyClray.png
zichuan lin
zichuan-lin
user
zichuan-lin
698557724ad556f294b7eb02
Jiafei Lyu
698557724ad556f294b7eb03
Yicheng Liu
claimed_verified
2026-02-17T15:50:17.151Z
68be47de9dd8d73e702c376b
/avatars/3bc26a294eecc63372ee77abb821fed7.svg
yichengliu
yiceason
user
yiceason
698557724ad556f294b7eb04
Zhicong Lu
698557724ad556f294b7eb05
Deheng Ye
698557724ad556f294b7eb06
Jie Jiang
ProAct: Agentic Lookahead in Interactive Environments
Existing Large Language Model (LLM) agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm. First, we introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search. By compressing complex search trees into concise, causal reasoning chains, the agent learns the logic of foresight without the computational overhead of inference-time search. Second, to further refine decision accuracy, we propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms like PPO and GRPO. By leveraging lightweight environment rollouts to calibrate value estimates, MC-Critic provides a low-variance signal that facilitates stable policy optimization without relying on expensive model-based value approximation. Experiments on both stochastic (e.g., 2048) and deterministic (e.g., Sokoban) environments demonstrate that ProAct significantly improves planning accuracy. Notably, a 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while demonstrating robust generalization to unseen environments. The codes and models are available at https://github.com/GreatX3/ProAct
ProAct enhances LLM agents' long-horizon planning by combining supervised fine-tuning with search-derived trajectories and a Monte-Carlo critic for improved policy optimization.
Large Language Model agents long-horizon planning lookahead reasoning two-stage training paradigm Grounded LookAhead Distillation Monte-Carlo Critic policy-gradient algorithms PPO GRPO environment-based search causal reasoning chains value estimation policy optimization stochastic environments deterministic environments