InteractComp: Evaluating Search Agents With Ambiguous Queries
Language agents have demonstrated remarkable potential in web search and
information retrieval. However, these search agents assume user queries are
complete and unambiguous, an assumption that diverges from reality where users
begin with incomplete queries requiring clarification through interaction. Yet
most agents lack interactive mechanisms during the search process, and existing
benchmarks cannot assess this capability. To address this gap, we introduce
InteractComp, a benchmark designed to evaluate whether search agents can
recognize query ambiguity and actively interact to resolve it during search.
Following the principle of easy to verify, interact to disambiguate, we
construct 210 expert-curated questions across 9 domains through a
target-distractor methodology that creates genuine ambiguity resolvable only
through interaction. Evaluation of 17 models reveals striking failure: the best
model achieves only 13.73% accuracy despite 71.50% with complete context,
exposing systematic overconfidence rather than reasoning deficits. Forced
interaction produces dramatic gains, demonstrating latent capability current
strategies fail to engage. Longitudinal analysis shows interaction capabilities
stagnated over 15 months while search performance improved seven-fold,
revealing a critical blind spot. This stagnation, coupled with the immediate
feedback inherent to search tasks, makes InteractComp a valuable resource for
both evaluating and training interaction capabilities in search agents. The
code is available at https://github.com/FoundationAgents/InteractComp.
690188f7646208eac0d1f482
Mingyi Deng
690188f7646208eac0d1f483
Lijun Huang
690188f7646208eac0d1f484
Yani Fan
690188f7646208eac0d1f485
Jiayi Zhang
claimed_verified
2025-10-29T12:43:11.330Z
65f40e83653c231cbaf7defe
/avatars/afa5ce72324112739e539865c9aee26b.svg
Jiayi Zhang
didiforhugface
user
didiforhugface
690188f7646208eac0d1f486
Fashen Ren
690188f7646208eac0d1f487
Jinyi Bai
690188f7646208eac0d1f488
Fuzhen Yang
690188f7646208eac0d1f489
Dayi Miao
690188f7646208eac0d1f48a
Zhaoyang Yu
claimed_verified
2025-11-20T08:50:34.594Z
640dc84b474aa6f89554d518
/avatars/9fcee1023ed5c6cddb3c342e19f18295.svg
Zhaoyang Yu
MoshiQAQ
user
MoshiQAQ
690188f7646208eac0d1f48b
Yifan Wu
690188f7646208eac0d1f48c
Yanfei Zhang
claimed_verified
2025-10-29T12:42:46.305Z
65ddeb1e344080c5bbc547e8
/avatars/8423d009aa8f3737b2309c64d1d71b65.svg
Yanfei Zhang
LovinYou
user
LovinYou
690188f7646208eac0d1f48d
Fengwei Teng
690188f7646208eac0d1f48e
Yingjia Wan
690188f7646208eac0d1f48f
Song Hu
690188f7646208eac0d1f490
Yude Li
690188f7646208eac0d1f491
Xin Jin
690188f7646208eac0d1f492
Conghao Hu
690188f7646208eac0d1f493
Haoyu Li
690188f7646208eac0d1f494
Qirui Fu
690188f7646208eac0d1f495
Tai Zhong
690188f7646208eac0d1f496
Xinyu Wang
claimed_verified
2025-10-30T14:41:50.255Z
65e5fa6a55ed97ae47253e67
/avatars/363ea68811f5aab72c47637385128346.svg
Xinyu Wang
W4ng1204
user
W4ng1204
690188f7646208eac0d1f497
Xiangru Tang
690188f7646208eac0d1f498
Nan Tang
690188f7646208eac0d1f499
Chenglin Wu
690188f7646208eac0d1f49a
Yuyu Luo
InteractComp: Evaluating Search Agents With Ambiguous Queries
Language agents have demonstrated remarkable potential in web search and
information retrieval. However, these search agents assume user queries are
complete and unambiguous, an assumption that diverges from reality where users
begin with incomplete queries requiring clarification through interaction. Yet
most agents lack interactive mechanisms during the search process, and existing
benchmarks cannot assess this capability. To address this gap, we introduce
InteractComp, a benchmark designed to evaluate whether search agents can
recognize query ambiguity and actively interact to resolve it during search.
Following the principle of easy to verify, interact to disambiguate, we
construct 210 expert-curated questions across 9 domains through a
target-distractor methodology that creates genuine ambiguity resolvable only
through interaction. Evaluation of 17 models reveals striking failure: the best
model achieves only 13.73% accuracy despite 71.50% with complete context,
exposing systematic overconfidence rather than reasoning deficits. Forced
interaction produces dramatic gains, demonstrating latent capability current
strategies fail to engage. Longitudinal analysis shows interaction capabilities
stagnated over 15 months while search performance improved seven-fold,
revealing a critical blind spot. This stagnation, coupled with the immediate
feedback inherent to search tasks, makes InteractComp a valuable resource for
both evaluating and training interaction capabilities in search agents. The
code is available at https://github.com/FoundationAgents/InteractComp.
InteractComp evaluates search agents' ability to recognize and resolve query ambiguity through interaction, revealing significant gaps in current models' capabilities.
search agents query ambiguity interaction mechanisms benchmark expert-curated questions target-distractor methodology accuracy overconfidence reasoning deficits longitudinal analysis