mHC: Manifold-Constrained Hyper-Connections
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
69561fbf832867f253525726
Zhenda Xie
admin_assigned
2026-01-02T16:36:55.290Z
6797ca96e9e2793006a15110
/avatars/2d393d6e5fc2e1a867f7fdd44e055a2f.svg
zhenda xie
Zhendaxie
user
Zhendaxie
69561fbf832867f253525727
Yixuan Wei
admin_assigned
2026-01-02T16:36:42.013Z
640efe54a92fedb0e84ee598
/avatars/cb1c6720525bfdbf59a9d3eac93a879e.svg
Yixuan Wei
EasonWei
user
EasonWei
69561fbf832867f253525728
Huanqi Cao
69561fbf832867f253525729
Chenggang Zhao
admin_assigned
2026-01-02T16:37:02.767Z
66053b1f9e3555d648b21c3d
/avatars/c8b33e7f702c4edb17add47f0eafe5e6.svg
Chenggang Zhao
LyricZ
user
LyricZ
69561fbf832867f25352572a
Chengqi Deng
69561fbf832867f25352572b
Jiashi Li
admin_assigned
2026-01-02T16:37:13.306Z
64fca5f28d50404bc42ca78a
/avatars/ae01ac0296d6ce1277dacb6894f570b8.svg
Jiashi Li
Beginlner
user
Beginlner
69561fbf832867f25352572c
Damai Dai
admin_assigned
2026-01-02T16:37:21.825Z
659389f8de82e1ef7b9a8b13
/avatars/896ed9f4cdbd317493b303d070b7e12a.svg
Damai Dai
DeepSeekDDM
user
DeepSeekDDM
69561fbf832867f25352572d
Huazuo Gao
admin_assigned
2026-01-02T16:37:28.878Z
64e370be59aa5366642ac329
/avatars/0fa1eb6ac6c1aeff3e65bc86a6617f64.svg
Huazuo Gao
gaohuazuo
user
gaohuazuo
69561fbf832867f25352572e
Jiang Chang
69561fbf832867f25352572f
Liang Zhao
69561fbf832867f253525730
Shangyan Zhou
admin_assigned
2026-01-02T16:37:39.269Z
654453e19b639f21e1d77d16
/avatars/079ec500c2ca7a31f6cb754b8c7ef065.svg
Shangyan Zhou
syzhou
user
syzhou
69561fbf832867f253525731
Zhean Xu
claimed_verified
2026-01-02T15:38:43.261Z
676d42cdf51984f73463fbeb
/avatars/156938eff95375b35db4b307739e9d89.svg
Zhean Xu
CyanicX
user
CyanicX
69561fbf832867f253525732
Zhengyan Zhang
admin_assigned
2026-01-02T16:36:24.246Z
65654ed2219af7f841640f27
/avatars/e6904b3479fc5e65ea1f752919ca8290.svg
Zhengyan Zhang
ZhengyanZhang
user
ZhengyanZhang
69561fbf832867f253525733
Wangding Zeng
admin_assigned
2026-01-02T16:37:50.603Z
659384dec0b1372b2e3c184d
/avatars/fdbff8012fe80d5f2e861cbfa2675125.svg
wangding zeng
zwd973-deepseek
user
zwd973-deepseek
69561fbf832867f253525734
Shengding Hu
69561fbf832867f253525735
Yuqing Wang
69561fbf832867f253525736
Jingyang Yuan
69561fbf832867f253525737
Lean Wang
admin_assigned
2026-01-02T16:39:15.341Z
650c509472afb1e60e6151ae
/avatars/c16ab5053a586819dc2b965303215ff7.svg
Lean Wang
AdaHousman
user
AdaHousman
69561fbf832867f253525738
Wenfeng Liang
mHC: Manifold-Constrained Hyper-Connections
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
Manifold-Constrained Hyper-Connections (mHC) stabilize and scale residual connection architectures by restoring identity mapping properties through manifold projection and infrastructure optimization.
Hyper-Connections (HC) Manifold-Constrained Hyper-Connections (mHC) residual connections residual stream width connectivity patterns identity mapping property training instability memory access overhead manifold projection infrastructure optimization scalability