BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language
  Models in Chinese
As large language models (LLMs) evolve into tool-using agents, the ability to
browse the web in real-time has become a critical yardstick for measuring their
reasoning and retrieval competence. Existing benchmarks such as BrowseComp
concentrate on English and overlook the linguistic, infrastructural, and
censorship-related complexities of other major information ecosystems -- most
notably Chinese. To address this gap, we introduce BrowseComp-ZH, a
high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents
on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning
11 diverse domains. Each question is reverse-engineered from a short,
objective, and easily verifiable answer (e.g., a date, number, or proper noun).
A two-stage quality control protocol is applied to strive for high question
difficulty and answer uniqueness. We benchmark over 20 state-of-the-art
language models and agentic search systems on our proposed BrowseComp-ZH.
Despite their strong conversational and retrieval capabilities, most models
struggle severely: a large number achieve accuracy rates below 10%, and only a
handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch,
reaches just 42.9%. These results demonstrate the considerable difficulty of
BrowseComp-ZH, where success demands not only effective retrieval strategies,
but also sophisticated reasoning and information reconciliation -- capabilities
that current models still struggle to master. Our dataset, construction
guidelines, and benchmark results have been publicly released at
https://github.com/PALIN2018/BrowseComp-ZH.
681d89e6d025518b321f67ce
Peilin Zhou
admin_assigned
2025-05-09T09:01:42.666Z
6673cf668d570d59b83511cc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/g9sJViXr1Cmx8A_is7Zgq.jpeg
Peilin Zhou
PALIN2018
user
PALIN2018
681d89e6d025518b321f67cf
Bruce Leon
681d89e6d025518b321f67d0
Xiang Ying
extracted_pending
2025-05-09T04:51:51.507Z
64489ca21d52a633c8f55aba
/avatars/5199a5e93161c61d14ec13f79dd8c2c5.svg
Xiang Ying
MindYing
user
MindYing
681d89e6d025518b321f67d1
Can Zhang
681d89e6d025518b321f67d2
Yifan Shao
681d89e6d025518b321f67d3
Qichen Ye
admin_assigned
2025-05-09T09:02:33.873Z
636dfa6193d9a0c987d41b73
/avatars/14396c8beb376b0d3c27a23fadaeb15e.svg
Qichen YE
yeeeqichen99
user
yeeeqichen99
681d89e6d025518b321f67d4
Dading Chong
681d89e6d025518b321f67d5
Zhiling Jin
admin_assigned
2025-05-09T09:02:48.180Z
66bc683f432c73a183ef787c
/avatars/9505a1e6131093a91d0454e50bcbba00.svg
Zhiling Jin
HawkFaust
user
HawkFaust
681d89e6d025518b321f67d6
Chenxuan Xie
681d89e6d025518b321f67d7
Meng Cao
681d89e6d025518b321f67d8
Yuxin Gu
681d89e6d025518b321f67d9
Sixin Hong
681d89e6d025518b321f67da
Jing Ren
681d89e6d025518b321f67db
Jian Chen
681d89e6d025518b321f67dc
Chao Liu
681d89e6d025518b321f67dd
Yining Hua
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language
  Models in Chinese
As large language models (LLMs) evolve into tool-using agents, the ability to
browse the web in real-time has become a critical yardstick for measuring their
reasoning and retrieval competence. Existing benchmarks such as BrowseComp
concentrate on English and overlook the linguistic, infrastructural, and
censorship-related complexities of other major information ecosystems -- most
notably Chinese. To address this gap, we introduce BrowseComp-ZH, a
high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents
on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning
11 diverse domains. Each question is reverse-engineered from a short,
objective, and easily verifiable answer (e.g., a date, number, or proper noun).
A two-stage quality control protocol is applied to strive for high question
difficulty and answer uniqueness. We benchmark over 20 state-of-the-art
language models and agentic search systems on our proposed BrowseComp-ZH.
Despite their strong conversational and retrieval capabilities, most models
struggle severely: a large number achieve accuracy rates below 10%, and only a
handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch,
reaches just 42.9%. These results demonstrate the considerable difficulty of
BrowseComp-ZH, where success demands not only effective retrieval strategies,
but also sophisticated reasoning and information reconciliation -- capabilities
that current models still struggle to master. Our dataset, construction
guidelines, and benchmark results have been publicly released at
https://github.com/PALIN2018/BrowseComp-ZH.
BrowseComp-ZH evaluates large language models on real-time Chinese web browsing tasks, highlighting challenges in retrieval and reasoning beyond existing English benchmarks.
large language models tool-using agents BrowseComp BrowseComp-ZH multi-hop questions quality control protocol agentic search systems effective retrieval strategies sophisticated reasoning information reconciliation