SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise
Spoken query retrieval is an important interaction mode in modern information retrieval. However, existing evaluation datasets are often limited to simple queries under constrained noise conditions, making them inadequate for assessing the robustness of spoken query retrieval systems under complex acoustic perturbations. To address this limitation, we present SQuTR, a robustness benchmark for spoken query retrieval that includes a large-scale dataset and a unified evaluation protocol. SQuTR aggregates 37,317 unique queries from six commonly used English and Chinese text retrieval datasets, spanning multiple domains and diverse query types. We synthesize speech using voice profiles from 200 real speakers and mix 17 categories of real-world environmental noise under controlled SNR levels, enabling reproducible robustness evaluation from quiet to highly noisy conditions. Under the unified protocol, we conduct large-scale evaluations on representative cascaded and end-to-end retrieval systems. Experimental results show that retrieval performance decreases as noise increases, with substantially different drops across systems. Even large-scale retrieval models struggle under extreme noise, indicating that robustness remains a critical bottleneck. Overall, SQuTR provides a reproducible testbed for benchmarking and diagnostic analysis, and facilitates future research on robustness in spoken query to text retrieval.
6992ceae50fb2c0be47839c1
Yuejie Li
claimed_verified
2026-02-24T09:51:49.640Z
66949e6d1b082600cf672bc3
/avatars/2c548cd1b8a1598933ac12f2d6a03e77.svg
liyuejie
xiaolijiayounuli
user
xiaolijiayounuli
6992ceae50fb2c0be47839c2
Ke Yang
6992ceae50fb2c0be47839c3
Yueying Hua
6992ceae50fb2c0be47839c4
Berlin Chen
claimed_verified
2026-02-16T15:36:03.590Z
69259bc3f1571271e94fa76b
/avatars/b872462030e064e2a8ddc284c5ebe67e.svg
berlin
berlin8587
user
berlin8587
6992ceae50fb2c0be47839c5
Jianhao Nie
6992ceae50fb2c0be47839c6
Yueping He
6992ceae50fb2c0be47839c7
Caixin Kang
claimed_verified
2026-03-17T08:21:41.446Z
63a286c7f30c4642278ed11a
/avatars/974fe4f3023791e97228c53330e9b4f3.svg
Tim Kang
Ukpkmkkk
user
Ukpkmkkk
SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise
Spoken query retrieval is an important interaction mode in modern information retrieval. However, existing evaluation datasets are often limited to simple queries under constrained noise conditions, making them inadequate for assessing the robustness of spoken query retrieval systems under complex acoustic perturbations. To address this limitation, we present SQuTR, a robustness benchmark for spoken query retrieval that includes a large-scale dataset and a unified evaluation protocol. SQuTR aggregates 37,317 unique queries from six commonly used English and Chinese text retrieval datasets, spanning multiple domains and diverse query types. We synthesize speech using voice profiles from 200 real speakers and mix 17 categories of real-world environmental noise under controlled SNR levels, enabling reproducible robustness evaluation from quiet to highly noisy conditions. Under the unified protocol, we conduct large-scale evaluations on representative cascaded and end-to-end retrieval systems. Experimental results show that retrieval performance decreases as noise increases, with substantially different drops across systems. Even large-scale retrieval models struggle under extreme noise, indicating that robustness remains a critical bottleneck. Overall, SQuTR provides a reproducible testbed for benchmarking and diagnostic analysis, and facilitates future research on robustness in spoken query to text retrieval.