Reasoning Language Model Inference Serving Unveiled: An Empirical Study
The reasoning large language model (RLLM) has been proven competitive in
solving complex reasoning tasks such as mathematics, coding, compared to
general LLM. However, the serving performance and behavior of RLLM remains
unexplored, which may undermine the deployment and utilization of RLLM in
real-world scenario. To close this gap, in this paper, we conduct a
comprehensive study of RLLM service. We first perform a pilot study on
comparing the serving performance between RLLM and traditional LLM and reveal
that there are several distinct differences regarding serving behavior: (1)
significant memory usage and fluctuations; (2) straggler requests; (3) adaptive
running time; (4) domain preference. Then we further investigate whether
existing inference optimization techniques are valid for RLLM. Our main
takeaways are that model quantization methods and speculative decoding can
improve service system efficiency with small compromise to RLLM accuracy, while
prefix caching, KV cache quantization may even degrade accuracy or serving
performance for small RLLM. Lastly, we conduct evaluation under real world
workload modeled by Gamma distribution to verify our findings. Empirical
results of real world workload evaluation across different dataset are aligned
with our main findings regarding RLLM serving. We hope our work can provide the
research community and industry with insights to advance RLLM inference
serving.
69035cf6274024e4098e32c0
Qi Li
claimed_verified
2025-10-30T14:10:32.144Z
69035aa0f9b8cf048fa94931
/avatars/1e6ed54685813e83cdaa703f0ecd42a8.svg
Qi Li
lqinfdim
user
lqinfdim
69035cf6274024e4098e32c1
Junpan Wu
69035cf6274024e4098e32c2
Xiang Liu
69035cf6274024e4098e32c3
Yuxin Wang
69035cf6274024e4098e32c4
Zeyu Li
69035cf6274024e4098e32c5
Zhenheng Tang
69035cf6274024e4098e32c6
Yuhan Chen
69035cf6274024e4098e32c7
Shaohuai Shi
69035cf6274024e4098e32c8
Xiaowen Chu
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
The reasoning large language model (RLLM) has been proven competitive in
solving complex reasoning tasks such as mathematics, coding, compared to
general LLM. However, the serving performance and behavior of RLLM remains
unexplored, which may undermine the deployment and utilization of RLLM in
real-world scenario. To close this gap, in this paper, we conduct a
comprehensive study of RLLM service. We first perform a pilot study on
comparing the serving performance between RLLM and traditional LLM and reveal
that there are several distinct differences regarding serving behavior: (1)
significant memory usage and fluctuations; (2) straggler requests; (3) adaptive
running time; (4) domain preference. Then we further investigate whether
existing inference optimization techniques are valid for RLLM. Our main
takeaways are that model quantization methods and speculative decoding can
improve service system efficiency with small compromise to RLLM accuracy, while
prefix caching, KV cache quantization may even degrade accuracy or serving
performance for small RLLM. Lastly, we conduct evaluation under real world
workload modeled by Gamma distribution to verify our findings. Empirical
results of real world workload evaluation across different dataset are aligned
with our main findings regarding RLLM serving. We hope our work can provide the
research community and industry with insights to advance RLLM inference
serving.
A study on the serving performance and behavior of reasoning large language models (RLLMs) reveals distinct differences from traditional LLMs and evaluates the effectiveness of various inference optimization techniques.
reasoning large language model RLLM serving performance serving behavior memory usage straggler requests adaptive running time domain preference model quantization speculative decoding prefix caching KV cache quantization Gamma distribution