VoxServe: Streaming-Centric Serving System for Speech Language Models
Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.
698264da9084cb4f0ecb5530
Keisuke Kamahori
claimed_verified
2026-02-04T12:29:49.344Z
6304ac1a412a1b9d381ca378
/avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
Keisuke Kamahori
kamahori
user
kamahori
698264da9084cb4f0ecb5531
Wei-Tzu Lee
698264da9084cb4f0ecb5532
Atindra Jha
698264da9084cb4f0ecb5533
Rohan Kadekodi
698264da9084cb4f0ecb5534
Stephanie Wang
698264da9084cb4f0ecb5535
Arvind Krishnamurthy
698264da9084cb4f0ecb5536
Baris Kasikci
VoxServe: Streaming-Centric Serving System for Speech Language Models
Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.
VoxServe is a unified serving system for Speech Language Models that enhances streaming performance through model-execution abstraction, streaming-aware scheduling, and asynchronous inference pipelines.
Speech Language Models streaming settings low latency high throughput streamability model-execution abstraction streaming-aware scheduling asynchronous inference pipeline