GISA: A Benchmark for General Information-Seeking Assistant
The advancement of large language models (LLMs) has significantly accelerated the development of search agents capable of autonomously gathering information through multi-turn web interactions. Various benchmarks have been proposed to evaluate such agents. However, existing benchmarks often construct queries backward from answers, producing unnatural tasks misaligned with real-world needs. Moreover, these benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries that reflect authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, offering gold-standard references for process-level supervision and imitation learning. Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30\% exact match score, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering. These findings highlight substantial room for future improvement.
698abf921b2dc6b37d61b16a
Yutao Zhu
698abf921b2dc6b37d61b16b
Xingshuo Zhang
698abf921b2dc6b37d61b16c
Maosen Zhang
698abf921b2dc6b37d61b16d
Jiajie Jin
698abf921b2dc6b37d61b16e
Liancheng Zhang
698abf921b2dc6b37d61b16f
Xiaoshuai Song
698abf921b2dc6b37d61b170
Kangzhi Zhao
698abf921b2dc6b37d61b171
Wencong Zeng
698abf921b2dc6b37d61b172
Ruiming Tang
698abf921b2dc6b37d61b173
Han Li
698abf921b2dc6b37d61b174
Ji-Rong Wen
698abf921b2dc6b37d61b175
Zhicheng Dou
claimed_verified
2026-02-10T09:05:14.771Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
douzc
user
douzc
GISA: A Benchmark for General Information-Seeking Assistant
The advancement of large language models (LLMs) has significantly accelerated the development of search agents capable of autonomously gathering information through multi-turn web interactions. Various benchmarks have been proposed to evaluate such agents. However, existing benchmarks often construct queries backward from answers, producing unnatural tasks misaligned with real-world needs. Moreover, these benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries that reflect authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, offering gold-standard references for process-level supervision and imitation learning. Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30\% exact match score, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering. These findings highlight substantial room for future improvement.
A new benchmark called GISA is introduced for evaluating information-seeking assistants, featuring human-crafted queries with structured answer formats and live updates to prevent memorization.
large language models search agents information-seeking assistants benchmarks deep reasoning information aggregation exact match score complex planning