Kimi Linear: An Expressive, Efficient Attention Architecture
We introduce Kimi Linear, a hybrid linear attention architecture that, for
the first time, outperforms full attention under fair comparisons across
various scenarios -- including short-context, long-context, and reinforcement
learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an
expressive linear attention module that extends Gated DeltaNet with a
finer-grained gating mechanism, enabling more effective use of limited
finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware
efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR)
transition matrices, which substantially reduces computation compared to the
general DPLR formulation while remaining more consistent with the classical
delta rule.
  We pretrain a Kimi Linear model with 3B activated parameters and 48B total
parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention
(MLA). Our experiments show that with an identical training recipe, Kimi Linear
outperforms full MLA with a sizeable margin across all evaluated tasks, while
reducing KV cache usage by up to 75% and achieving up to 6 times decoding
throughput for a 1M context. These results demonstrate that Kimi Linear can be
a drop-in replacement for full attention architectures with superior
performance and efficiency, including tasks with longer input and output
lengths.
  To support further research, we open-source the KDA kernel and vLLM
implementations, and release the pre-trained and instruction-tuned model
checkpoints.
690414082c556835fa67ef77
Kimi Team
690414082c556835fa67ef78
Yu Zhang
claimed_verified
2025-11-03T20:56:53.451Z
638f5839f6de4b9e7e1627fb
https://cdn-avatars.huggingface.co/v1/production/uploads/638f5839f6de4b9e7e1627fb/6QGkrqRag6-GnH9k60Oil.jpeg
Yu Zhang
yzhangcs
user
yzhangcs
690414082c556835fa67ef79
Zongyu Lin
690414082c556835fa67ef7a
Xingcheng Yao
690414082c556835fa67ef7b
Jiaxi Hu
claimed_verified
2025-11-06T13:45:47.208Z
665dc35752ff9daa9ba5a4ed
/avatars/df8b01879d97e599b610fa51414d3a18.svg
Hu Jiaxi
Jiaxihu2
user
Jiaxihu2
690414082c556835fa67ef7c
Fanqing Meng
690414082c556835fa67ef7d
Chengyin Liu
690414082c556835fa67ef7e
Xin Men
690414082c556835fa67ef7f
Songlin Yang
690414082c556835fa67ef80
Zhiyuan Li
claimed_verified
2025-10-31T14:27:40.234Z
65e98b74b8dbf436e6585106
https://cdn-avatars.huggingface.co/v1/production/uploads/65e98b74b8dbf436e6585106/NIJAj5pTFduIucifFe80a.jpeg
LiZhiyuan
zhiyuan8
user
zhiyuan8
690414082c556835fa67ef81
Wentao Li
690414082c556835fa67ef82
Enzhe Lu
690414082c556835fa67ef83
Weizhou Liu
690414082c556835fa67ef84
Yanru Chen
690414082c556835fa67ef85
Weixin Xu
claimed_verified
2025-11-28T13:49:12.573Z
63a27cb03c003e409325dda2
https://cdn-avatars.huggingface.co/v1/production/uploads/1671608965116-63a27cb03c003e409325dda2.png
Weixin Xu
xuwwwux
user
xuwwwux
690414082c556835fa67ef86
Longhui Yu
690414082c556835fa67ef87
Yejie Wang
claimed_verified
2025-11-03T20:56:49.396Z
6342dc0ee2647466b42918ab
/avatars/a80a3df2f67410662bf9681ed8834b17.svg
Yejie Wang
banksy235
user
banksy235
690414082c556835fa67ef88
Yu Fan
690414082c556835fa67ef89
Longguang Zhong
claimed_verified
2025-11-03T20:56:56.804Z
62b6d20416ff90e6198301b6
https://cdn-avatars.huggingface.co/v1/production/uploads/1656148456743-noauth.png
Longguang Zhong
GGLS
user
GGLS
690414082c556835fa67ef8a
Enming Yuan
690414082c556835fa67ef8b
Dehao Zhang
690414082c556835fa67ef8c
Yizhi Zhang
690414082c556835fa67ef8d
T. Y. Liu
690414082c556835fa67ef8e
Haiming Wang
690414082c556835fa67ef8f
Shengjun Fang
690414082c556835fa67ef90
Weiran He
690414082c556835fa67ef91
Shaowei Liu
690414082c556835fa67ef92
Yiwei Li
690414082c556835fa67ef93
Jianlin Su
690414082c556835fa67ef94
Jiezhong Qiu
690414082c556835fa67ef95
Bo Pang
690414082c556835fa67ef96
Junjie Yan
690414082c556835fa67ef97
Zhejun Jiang
690414082c556835fa67ef98
Weixiao Huang
690414082c556835fa67ef99
Bohong Yin
690414082c556835fa67ef9a
Jiacheng You
690414082c556835fa67ef9b
Chu Wei
claimed_verified
2026-02-03T14:00:02.205Z
635ddec594e5b275ca7941e8
/avatars/28ebfaee74d31e1de020a3ae735a4c1b.svg
Chu Wei
courage17340
user
courage17340
690414082c556835fa67ef9c
Zhengtao Wang
690414082c556835fa67ef9d
Chao Hong
690414082c556835fa67ef9e
Yutian Chen
690414082c556835fa67ef9f
Guanduo Chen
690414082c556835fa67efa0
Yucheng Wang
690414082c556835fa67efa1
Huabin Zheng
690414082c556835fa67efa2
Feng Wang
690414082c556835fa67efa3
Yibo Liu
690414082c556835fa67efa4
Mengnan Dong
690414082c556835fa67efa5
Zheng Zhang
690414082c556835fa67efa6
Siyuan Pan
690414082c556835fa67efa7
Wenhao Wu
690414082c556835fa67efa8
Yuhao Wu
claimed_verified
2025-11-03T20:56:46.103Z
63369da91ba5d5ece24118a4
/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg
wuyuhao
mozhu
user
mozhu
690414082c556835fa67efa9
Longyu Guan
690414082c556835fa67efaa
Jiawen Tao
690414082c556835fa67efab
Guohong Fu
690414082c556835fa67efac
Xinran Xu
690414082c556835fa67efad
Yuzhi Wang
690414082c556835fa67efae
Guokun Lai
690414082c556835fa67efaf
Yuxin Wu
690414082c556835fa67efb0
Xinyu Zhou
690414082c556835fa67efb1
Zhilin Yang
690414082c556835fa67efb2
Yulun Du
claimed_verified
2026-02-03T10:10:14.682Z
6340f31fb78ed99eab04ce33
/avatars/2e7fcbf0233bdc0bc9a3f4603fd8bf90.svg
Du
Yulun
user
Yulun
Kimi Linear: An Expressive, Efficient Attention Architecture
We introduce Kimi Linear, a hybrid linear attention architecture that, for
the first time, outperforms full attention under fair comparisons across
various scenarios -- including short-context, long-context, and reinforcement
learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an
expressive linear attention module that extends Gated DeltaNet with a
finer-grained gating mechanism, enabling more effective use of limited
finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware
efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR)
transition matrices, which substantially reduces computation compared to the
general DPLR formulation while remaining more consistent with the classical
delta rule.
  We pretrain a Kimi Linear model with 3B activated parameters and 48B total
parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention
(MLA). Our experiments show that with an identical training recipe, Kimi Linear
outperforms full MLA with a sizeable margin across all evaluated tasks, while
reducing KV cache usage by up to 75% and achieving up to 6 times decoding
throughput for a 1M context. These results demonstrate that Kimi Linear can be
a drop-in replacement for full attention architectures with superior
performance and efficiency, including tasks with longer input and output
lengths.
  To support further research, we open-source the KDA kernel and vLLM
implementations, and release the pre-trained and instruction-tuned model
checkpoints.
Kimi Linear, a hybrid linear attention architecture, outperforms full attention in various scenarios with improved efficiency and performance, using Kimi Delta Attention and Multi-Head Latent Attention.
linear attention Kimi Delta Attention Gated DeltaNet finite-state RNN memory chunkwise algorithm Diagonal-Plus-Low-Rank MLA KV cache decoding throughput vLLM