Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.
69b11e511f7902d47f6dfb49
Maike Züfle
69b11e511f7902d47f6dfb4a
Sara Papi
claimed_verified
2026-03-11T08:28:16.275Z
66309b3833ccd9e68c5d5171
https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg
Sara Papi
spapi
user
spapi
69b11e511f7902d47f6dfb4b
Fabian Retkowski
69b11e511f7902d47f6dfb4c
Szymon Mazurek
69b11e511f7902d47f6dfb4d
Marek Kasztelnik
69b11e511f7902d47f6dfb4e
Alexander Waibel
69b11e511f7902d47f6dfb4f
Luisa Bentivogli
69b11e511f7902d47f6dfb50
Jan Niehues
Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompts, which may not reflect real-world scenarios where users interact with speech. To address this gap, we introduce DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to pair with any existing benchmark for realistic evaluation of SLLMs under spoken instruction conditions. Spanning 9 tasks and 11 languages, it provides 10 prompt variants per task-language pair, across five styles. Using DOWIS, we benchmark state-of-the-art SLLMs, analyzing the interplay between prompt modality, style, language, and task type. Results show that text prompts consistently outperform spoken prompts, particularly for low-resource and cross-lingual settings. Only for tasks with speech output, spoken prompts do close the gap, highlighting the need for speech-based prompting in SLLM evaluation.
Speech Large Language Models are evaluated using a new multilingual dataset that pairs spoken and written prompts across multiple tasks and languages, revealing that text prompts generally outperform spoken prompts except in speech-output tasks.
Speech Large Language Models multilingual dataset spoken prompts written prompts benchmarking prompt modality task type language cross-lingual settings speech-output tasks