Repurposing Synthetic Data for Fine-grained Search Agent Supervision
LLM-based search agents are increasingly trained on entity-centric synthetic
data to solve complex, knowledge-intensive tasks. However, prevailing training
methods like Group Relative Policy Optimization (GRPO) discard this rich entity
information, relying instead on sparse, outcome-based rewards. This critical
limitation renders them unable to distinguish informative "near-miss"
samples-those with substantially correct reasoning but a flawed final
answer-from complete failures, thus discarding valuable learning signals. We
address this by leveraging the very entities discarded during training. Our
empirical analysis reveals a strong positive correlation between the number of
ground-truth entities identified during an agent's reasoning process and final
answer accuracy. Building on this insight, we introduce Entity-aware Group
Relative Policy Optimization (E-GRPO), a novel framework that formulates a
dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect
samples proportional to their entity match rate, enabling the model to
effectively learn from these "near-misses". Experiments on diverse
question-answering (QA) and deep research benchmarks show that E-GRPO
consistently and significantly outperforms the GRPO baseline. Furthermore, our
analysis reveals that E-GRPO not only achieves superior accuracy but also
induces more efficient reasoning policies that require fewer tool calls,
demonstrating a more effective and sample-efficient approach to aligning search
agents.
69017e04646208eac0d1f42e
Yida Zhao
claimed_verified
2025-10-29T12:43:33.267Z
66e4019518a1920fb7ca19d7
/avatars/4f8be2f3244239510235c2ec6e8d38a6.svg
Yida Zhao
zhaoyd
user
zhaoyd
69017e04646208eac0d1f42f
Kuan Li
69017e04646208eac0d1f430
Xixi Wu
claimed_verified
2025-10-29T12:43:30.221Z
6622132f63598534f96ca29d
/avatars/34e61fc3101f8ebce1ef7041f761e108.svg
Xixi Wu
xxwu
user
xxwu
69017e04646208eac0d1f431
Liwen Zhang
69017e04646208eac0d1f432
Dingchu Zhang
69017e04646208eac0d1f433
Baixuan Li
69017e04646208eac0d1f434
Maojia Song
69017e04646208eac0d1f435
Zhuo Chen
69017e04646208eac0d1f436
Chenxi Wang
69017e04646208eac0d1f437
Xinyu Wang
69017e04646208eac0d1f438
Kewei Tu
69017e04646208eac0d1f439
Pengjun Xie
69017e04646208eac0d1f43a
Jingren Zhou
69017e04646208eac0d1f43b
Yong Jiang
Repurposing Synthetic Data for Fine-grained Search Agent Supervision
LLM-based search agents are increasingly trained on entity-centric synthetic
data to solve complex, knowledge-intensive tasks. However, prevailing training
methods like Group Relative Policy Optimization (GRPO) discard this rich entity
information, relying instead on sparse, outcome-based rewards. This critical
limitation renders them unable to distinguish informative "near-miss"
samples-those with substantially correct reasoning but a flawed final
answer-from complete failures, thus discarding valuable learning signals. We
address this by leveraging the very entities discarded during training. Our
empirical analysis reveals a strong positive correlation between the number of
ground-truth entities identified during an agent's reasoning process and final
answer accuracy. Building on this insight, we introduce Entity-aware Group
Relative Policy Optimization (E-GRPO), a novel framework that formulates a
dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect
samples proportional to their entity match rate, enabling the model to
effectively learn from these "near-misses". Experiments on diverse
question-answering (QA) and deep research benchmarks show that E-GRPO
consistently and significantly outperforms the GRPO baseline. Furthermore, our
analysis reveals that E-GRPO not only achieves superior accuracy but also
induces more efficient reasoning policies that require fewer tool calls,
demonstrating a more effective and sample-efficient approach to aligning search
agents.
Entity-aware Group Relative Policy Optimization (E-GRPO) enhances search agents by incorporating entity information into the reward function, improving accuracy and efficiency in complex, knowledge-intensive tasks.
Group Relative Policy Optimization (GRPO) Entity-aware Group Relative Policy Optimization (E-GRPO) entity-centric synthetic data entity match rate near-misses question-answering (QA) deep research benchmarks tool calls