How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.
69ccc9bd3f84a0971be66c48
Ke-Han Lu
claimed_verified
2026-04-01T08:34:14.602Z
61001ae1eb9f3f5d28d1772c
/avatars/25314decfaa5c19d1136b37338683f48.svg
Ke-Han Lu
kehanlu
user
kehanlu
69ccc9bd3f84a0971be66c49
Szu-Wei Fu
69ccc9bd3f84a0971be66c4a
Chao-Han Huck Yang
claimed_verified
2026-04-16T20:22:29.295Z
629e1b71bb6419817ed7566c
https://cdn-avatars.huggingface.co/v1/production/uploads/629e1b71bb6419817ed7566c/0ZCt-11eQtRDCOk9AozOp.jpeg
Huck Yang
huckiyang
user
huckiyang
69ccc9bd3f84a0971be66c4b
Zhehuai Chen
69ccc9bd3f84a0971be66c4c
Sung-Feng Huang
69ccc9bd3f84a0971be66c4d
Chih-Kai Yang
69ccc9bd3f84a0971be66c4e
Yi-Cheng Lin
69ccc9bd3f84a0971be66c4f
Chi-Yuan Hsiao
69ccc9bd3f84a0971be66c50
Wenze Ren
69ccc9bd3f84a0971be66c51
En-Pei Hu
69ccc9bd3f84a0971be66c52
Yu-Han Huang
69ccc9bd3f84a0971be66c53
An-Yu Cheng
69ccc9bd3f84a0971be66c54
Cheng-Han Chiang
69ccc9bd3f84a0971be66c55
Yu Tsao
69ccc9bd3f84a0971be66c56
Yu-Chiang Frank Wang
69ccc9bd3f84a0971be66c57
Hung-yi Lee
How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.