Efficient Reasoning on the Edge
Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large context requirements make them impractical for edge deployment. These challenges include high token generation costs, large KV-cache footprints, and inefficiencies when distilling reasoning capabilities into smaller models for mobile devices. Existing approaches often rely on distilling reasoning traces from larger models into smaller models, which are verbose and stylistically redundant, undesirable for on-device inference. In this work, we propose a lightweight approach to enable reasoning in small LLMs using LoRA adapters combined with supervised fine-tuning. We further introduce budget forcing via reinforcement learning on these adapters, significantly reducing response length with minimal accuracy loss. To address memory-bound decoding, we exploit parallel test-time scaling, improving accuracy at minor latency increase. Finally, we present a dynamic adapter-switching mechanism that activates reasoning only when needed and a KV-cache sharing strategy during prompt encoding, reducing time-to-first-token for on-device inference. Experiments on Qwen2.5-7B demonstrate that our method achieves efficient, accurate reasoning under strict resource constraints, making LLM reasoning practical for mobile scenarios. Videos demonstrating our solution running on mobile devices are available on our project page.
69ba14c95d2b9c28ea09f107
Yelysei Bondarenko
claimed_verified
2026-03-26T14:04:28.226Z
649571182f09f73fc9c71340
/avatars/5b51bbeac91b6a53712e61a30b670e67.svg
Bondarenko
Yelysei
user
Yelysei
69ba14c95d2b9c28ea09f108
Thomas Hehn
69ba14c95d2b9c28ea09f109
Rob Hesselink
69ba14c95d2b9c28ea09f10a
Romain Lepert
69ba14c95d2b9c28ea09f10b
Fabio Valerio Massoli
claimed_verified
2026-03-19T13:02:49.866Z
69ba8344fd2b29bfd8c5cf90
/avatars/270892399c7c83b351fc27b0c71dbf09.svg
Fabio Valerio Massoli
fvmassoli
user
fvmassoli
69ba14c95d2b9c28ea09f10c
Evgeny Mironov
69ba14c95d2b9c28ea09f10d
Leyla Mirvakhabova
69ba14c95d2b9c28ea09f10e
Tribhuvanesh Orekondy
claimed_verified
2026-03-19T13:02:51.968Z
67ae1afcbdb93544392f28f5
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GMGpkOUEf6DSMP-r-wPOw.png
Tribhuvanesh Orekondy
torekond
user
torekond
69ba14c95d2b9c28ea09f10f
Spyridon Stasis
69ba14c95d2b9c28ea09f110
Andrey Kuzmin
69ba14c95d2b9c28ea09f111
Anna Kuzina
69ba14c95d2b9c28ea09f112
Markus Nagel
69ba14c95d2b9c28ea09f113
Ankita Nayak
admin_assigned
2026-03-18T10:11:34.310Z
687611dd5a6752969a50d086
/avatars/a51b59b2fac77ac8a3f0892cde61e674.svg
ANKITA NAYAK
nayakankita
user
nayakankita
69ba14c95d2b9c28ea09f114
Corrado Rainone
admin_assigned
2026-03-18T10:11:27.407Z
669e707ec517d804cfce91c5
/avatars/fc18e64e100cbf28763db04b44242747.svg
Corrado Rainone
crainone
user
crainone
69ba14c95d2b9c28ea09f115
Ork de Rooij
admin_assigned
2026-03-18T10:11:18.666Z
68e7940f7b9f8592b18752b2
/avatars/f7a4b5cc5c3a499895591342e7e7c1dc.svg
Ork de Rooij
oderooij
user
oderooij
69ba14c95d2b9c28ea09f116
Paul N Whatmough
69ba14c95d2b9c28ea09f117
Arash Behboodi
admin_assigned
2026-03-18T10:11:08.863Z
66dea675fd9b7f48c8067a70
/avatars/c02c46bfff0ee1af55f2ccd4ba3ab9ea.svg
Arash Behboodi
arashbehboodi
user
arashbehboodi
69ba14c95d2b9c28ea09f118
Babak Ehteshami Bejnordi
admin_assigned
2026-03-18T10:11:02.359Z
645ce510f5760d1530d322cb
/avatars/139d1b5b2e6a83ef39298e4dc17258ff.svg
Babak Ehteshami Bejnordi
babakint
user
babakint
Efficient Reasoning on the Edge
Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large context requirements make them impractical for edge deployment. These challenges include high token generation costs, large KV-cache footprints, and inefficiencies when distilling reasoning capabilities into smaller models for mobile devices. Existing approaches often rely on distilling reasoning traces from larger models into smaller models, which are verbose and stylistically redundant, undesirable for on-device inference. In this work, we propose a lightweight approach to enable reasoning in small LLMs using LoRA adapters combined with supervised fine-tuning. We further introduce budget forcing via reinforcement learning on these adapters, significantly reducing response length with minimal accuracy loss. To address memory-bound decoding, we exploit parallel test-time scaling, improving accuracy at minor latency increase. Finally, we present a dynamic adapter-switching mechanism that activates reasoning only when needed and a KV-cache sharing strategy during prompt encoding, reducing time-to-first-token for on-device inference. Experiments on Qwen2.5-7B demonstrate that our method achieves efficient, accurate reasoning under strict resource constraints, making LLM reasoning practical for mobile scenarios. Videos demonstrating our solution running on mobile devices are available on our project page.
Lightweight reasoning in small language models is enabled through LoRA adapters, budget forcing via reinforcement learning, parallel test-time scaling, and dynamic adapter switching under strict resource constraints.
large language models chain-of-thought reasoning LoRA adapters supervised fine-tuning reinforcement learning KV-cache parallel test-time scaling dynamic adapter-switching prompt encoding time-to-first-token