Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. Specifically, we identify two central architectural factors: depth-width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth-width ratios, with the key finding that although deep-thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy-latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy-latency frontier. In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy-efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5% average accuracy, 1.3x/1.9x lower latency, and 18.7x/45.6x higher throughput compared to Qwen3-1.7B/0.6B, respectively.
69274da1243b2216fb75ccd2
Yonggan Fu
69274da1243b2216fb75ccd3
Xin Dong
claimed_verified
2026-01-12T10:37:37.038Z
6092aaac134b77647f624bce
https://cdn-avatars.huggingface.co/v1/production/uploads/6092aaac134b77647f624bce/HGc1cJo7iOFJMKpZWrYJ0.jpeg
Xin Dong
SimonX
user
SimonX
69274da1243b2216fb75ccd4
Shizhe Diao
69274da1243b2216fb75ccd5
Matthijs Van keirsbilck
69274da1243b2216fb75ccd6
Hanrong Ye
69274da1243b2216fb75ccd7
Wonmin Byeon
69274da1243b2216fb75ccd8
Yashaswi Karnati
69274da1243b2216fb75ccd9
Lucas Liebenwein
69274da1243b2216fb75ccda
Hannah Zhang
69274da1243b2216fb75ccdb
Nikolaus Binder
69274da1243b2216fb75ccdc
Maksim Khadkevich
69274da1243b2216fb75ccdd
Alexander Keller
69274da1243b2216fb75ccde
Jan Kautz
69274da1243b2216fb75ccdf
Yingyan Celine Lin
69274da1243b2216fb75cce0
Pavlo Molchanov
claimed_verified
2026-01-22T08:48:01.447Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
pmolchanov
user
pmolchanov
Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. Specifically, we identify two central architectural factors: depth-width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth-width ratios, with the key finding that although deep-thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy-latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy-latency frontier. In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy-efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5% average accuracy, 1.3x/1.9x lower latency, and 18.7x/45.6x higher throughput compared to Qwen3-1.7B/0.6B, respectively.
The study identifies key architectural factors and efficient operators to optimize small language models for real-device latency, introducing the Nemotron-Flash family for improved accuracy and efficiency.
small language models SLMs parameter-efficient fine-tuning depth-width ratios operator choices efficient attention alternatives evolutionary search framework weight normalization