Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search
Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.
69d861b979548974da94655c
Chuzhan Hao
claimed_verified
2026-04-10T07:43:08.214Z
64ae631b58bd9e9cc2f5a749
/avatars/ce6426ec3bdb618a9e449297e7f147e0.svg
Chuzhan HAO
Chuzhan
user
Chuzhan
69d861b979548974da94655d
Wenfeng Feng
69d861b979548974da94655e
Guochao Jiang
claimed_verified
2026-04-10T07:43:11.658Z
644a1dbb9c340e5e1e713153
/avatars/21cb93ad067a798a39829ef7e67c70b8.svg
JGC
Nothing2Say
user
Nothing2Say
69d861b979548974da94655f
Guofeng Quan
69d861b979548974da946560
Guohua Liu
69d861b979548974da946561
Yuewei Zhang
Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search
Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.
A novel hierarchical experience framework improves reinforcement learning-based search agents by transforming raw reasoning trajectories into structured knowledge, enhancing both performance and training stability in complex reasoning tasks.
reinforcement learning large language models external search engines stochastic exploration outcome rewards hierarchical experience contrastive analysis multi-level clustering reasoning trajectories experience-aligned training agentic search mathematical reasoning