KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices
The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to its training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exact doubly stochastic residual matrices; 2) mHC incurs a prohibitive O(n^3C) parameter complexity with n as the width of the residual stream and C as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, O left( nC cdot n! right). To address both challenges, we propose KromHC, which uses the Kronecker products of smaller doubly stochastic matrices to parametrize the residual matrix in mHC. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to O(n^2C). Comprehensive experiments demonstrate that KromHC matches or even outperforms state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters. The code is available at https://github.com/wz1119/KromHC.
697c9eaa6676f933227060a3
Wuyang Zhou
claimed_verified
2026-01-30T13:31:13.092Z
67884550b5c0ef3754135924
/avatars/01bc97f18c4c2f35accfcf3b8a4fb5e0.svg
Wuyang Zhou
WuyangZzzz
user
WuyangZzzz
697c9eaa6676f933227060a4
Yuxuan Gu
claimed_verified
2026-02-02T16:59:04.759Z
676e805606c8364dc66aae8e
/avatars/460acfae9db74c3612b055c412aab738.svg
Yuxuan Gu
yuxuan-gu
user
yuxuan-gu
697c9eaa6676f933227060a5
Giorgos Iacovides
claimed_verified
2026-02-02T16:59:02.778Z
64b7c9828c632fbca94f700b
/avatars/5ae8a31efa83c70c28c9a10a7f077f55.svg
Giorgos Iacovides
georgeiac00
user
georgeiac00
697c9eaa6676f933227060a6
Danilo Mandic
KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices
The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to its training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exact doubly stochastic residual matrices; 2) mHC incurs a prohibitive O(n^3C) parameter complexity with n as the width of the residual stream and C as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, O left( nC cdot n! right). To address both challenges, we propose KromHC, which uses the Kronecker products of smaller doubly stochastic matrices to parametrize the residual matrix in mHC. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to O(n^2C). Comprehensive experiments demonstrate that KromHC matches or even outperforms state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters. The code is available at https://github.com/wz1119/KromHC.
KromHC addresses training instability and scalability issues in hyper-connections by using Kronecker products to parametrize residual matrices with reduced parameter complexity.
Hyper-Connections Manifold-Constrained Hyper-Connections Birkhoff polytope Sinkhorn-Knopp algorithm doubly stochastic matrices Birkhoff-von-Neumann theorem Kronecker products tensorized residual stream parameter complexity