On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving attributes like speaker and emotion, serving as foundation models for spoken dialogue. In prior literature, these models are often evaluated using ``global token perplexity'', which directly applies the text perplexity formulation to speech tokens. However, this practice overlooks fundamental differences between speech and text modalities, possibly leading to an underestimation of the speech characteristics. In this work, we propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity. We demonstrate that the proposed evaluations more faithfully reflect perceived generation quality, as evidenced by stronger correlations with human-rated mean opinion scores (MOS). When assessed under the new metrics, the relative performance landscape of spoken language models is reshaped, revealing a significantly reduced gap between the best-performing model and the human topline. Together, these results suggest that appropriate evaluation is critical for accurately assessing progress in spoken language modeling.
6965d559fc8c4ecc02c7f9fa
Jeff Chan-Jan Sju
6965d559fc8c4ecc02c7f9fb
Liang-Hsuan Tseng
6965d559fc8c4ecc02c7f9fc
Yi-Cheng Lin
6965d559fc8c4ecc02c7f9fd
Yen-Chun Kuo
6965d559fc8c4ecc02c7f9fe
Ju-Chieh Chou
6965d559fc8c4ecc02c7f9ff
Kai-Wei Chang
6965d559fc8c4ecc02c7fa00
Hung-yi Lee
6965d559fc8c4ecc02c7fa01
Carlos Busso
On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving attributes like speaker and emotion, serving as foundation models for spoken dialogue. In prior literature, these models are often evaluated using ``global token perplexity'', which directly applies the text perplexity formulation to speech tokens. However, this practice overlooks fundamental differences between speech and text modalities, possibly leading to an underestimation of the speech characteristics. In this work, we propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity. We demonstrate that the proposed evaluations more faithfully reflect perceived generation quality, as evidenced by stronger correlations with human-rated mean opinion scores (MOS). When assessed under the new metrics, the relative performance landscape of spoken language models is reshaped, revealing a significantly reduced gap between the best-performing model and the human topline. Together, these results suggest that appropriate evaluation is critical for accurately assessing progress in spoken language modeling.
Speech models trained on raw audio can generate appropriate content while maintaining speaker and emotion attributes, but traditional text-based evaluation methods underestimate speech characteristics; new evaluation approaches better correlate with human perception.
generative spoken language models raw audio speech tokens global token perplexity likelihood-based evaluation generative-based evaluation mean opinion scores human topline