MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
Scaling Large Language Models (LLMs) typically relies on increasing the number of parameters or test-time computations to boost performance. However, these strategies are impractical for edge device deployment due to limited RAM and NPU resources. Despite hardware constraints, deploying performant LLM on edge devices such as smartphone remains crucial for user experience. To address this, we propose MeKi (Memory-based Expert Knowledge Injection), a novel system that scales LLM capacity via storage space rather than FLOPs. MeKi equips each Transformer layer with token-level memory experts that injects pre-stored semantic knowledge into the generation process. To bridge the gap between training capacity and inference efficiency, we employ a re-parameterization strategy to fold parameter matrices used during training into a compact static lookup table. By offloading the knowledge to ROM, MeKi decouples model capacity from computational cost, introducing zero inference latency overhead. Extensive experiments demonstrate that MeKi significantly outperforms dense LLM baselines with identical inference speed, validating the effectiveness of memory-based scaling paradigm for on-device LLMs. Project homepage is at https://github.com/ningding-o/MeKi.
69836e4437bca54cd0587d92
Ning Ding
admin_assigned
2026-02-05T16:19:05.424Z
68b80d195038aabf1e0cd726
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8Lc3EPae0Kp_Yig74uMow.png
Ning Ding
Ning-Ding
user
Ning-Ding
69836e4437bca54cd0587d93
Fangcheng Liu
claimed_verified
2026-02-06T18:56:08.198Z
64b78295479b934973e2c40e
/avatars/9213e385964132fa50859264a838d891.svg
liu
Fangcheng2
user
Fangcheng2
69836e4437bca54cd0587d94
Kyungrae Kim
69836e4437bca54cd0587d95
Linji Hao
69836e4437bca54cd0587d96
Kyeng-Hun Lee
69836e4437bca54cd0587d97
Hyeonmok Ko
69836e4437bca54cd0587d98
Yehui Tang
MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling
Scaling Large Language Models (LLMs) typically relies on increasing the number of parameters or test-time computations to boost performance. However, these strategies are impractical for edge device deployment due to limited RAM and NPU resources. Despite hardware constraints, deploying performant LLM on edge devices such as smartphone remains crucial for user experience. To address this, we propose MeKi (Memory-based Expert Knowledge Injection), a novel system that scales LLM capacity via storage space rather than FLOPs. MeKi equips each Transformer layer with token-level memory experts that injects pre-stored semantic knowledge into the generation process. To bridge the gap between training capacity and inference efficiency, we employ a re-parameterization strategy to fold parameter matrices used during training into a compact static lookup table. By offloading the knowledge to ROM, MeKi decouples model capacity from computational cost, introducing zero inference latency overhead. Extensive experiments demonstrate that MeKi significantly outperforms dense LLM baselines with identical inference speed, validating the effectiveness of memory-based scaling paradigm for on-device LLMs. Project homepage is at https://github.com/ningding-o/MeKi.
MeKi enables efficient large language model deployment on edge devices by injecting pre-stored semantic knowledge through token-level memory experts and re-parameterization techniques.
Transformer layer token-level memory experts re-parameterization strategy parameter matrices lookup table ROM memory-based scaling paradigm