Stronger Normalization-Free Transformers
Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce Derf(x) = erf(αx + s), where erf(x) is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including vision (image recognition and generation), speech representation, and DNA sequence modeling. Our findings suggest that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.
693b85ec9874a2a5e4ffb2ee
Mingzhi Chen
admin_assigned
2025-12-12T13:13:32.371Z
64bfbbf06fc8c03398804a0c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Ij2peTEyoGnx-J1wN8P-p.png
Mingzhi Chen
Fishloong
user
Fishloong
693b85ec9874a2a5e4ffb2ef
Taiming Lu
admin_assigned
2025-12-12T13:13:39.570Z
656a9b9f9496f21be8271f1b
https://cdn-avatars.huggingface.co/v1/production/uploads/656a9b9f9496f21be8271f1b/oOcpLo9-anHJJ6kJ0BF1P.jpeg
TaiMing
TaiMingLu
user
TaiMingLu
693b85ec9874a2a5e4ffb2f0
Jiachen Zhu
693b85ec9874a2a5e4ffb2f1
Mingjie Sun
693b85ec9874a2a5e4ffb2f2
Zhuang Liu
Stronger Normalization-Free Transformers
Although normalization layers have long been viewed as indispensable components of deep learning architectures, the recent introduction of Dynamic Tanh (DyT) has demonstrated that alternatives are possible. The point-wise function DyT constrains extreme values for stable convergence and reaches normalization-level performance; this work seeks further for function designs that can surpass it. We first study how the intrinsic properties of point-wise functions influence training and performance. Building on these findings, we conduct a large-scale search for a more effective function design. Through this exploration, we introduce Derf(x) = erf(αx + s), where erf(x) is the rescaled Gaussian cumulative distribution function, and identify it as the most performant design. Derf outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains, including vision (image recognition and generation), speech representation, and DNA sequence modeling. Our findings suggest that the performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity. Its simplicity and stronger performance make Derf a practical choice for normalization-free Transformer architectures.
Derf, a novel point-wise normalization function, outperforms existing alternatives across various domains, enhancing generalization without increased fitting capacity.
Dynamic Tanh point-wise functions intrinsic properties large-scale search Derf rescaled Gaussian cumulative distribution function LayerNorm RMSNorm vision image recognition image generation speech representation DNA sequence modeling normalization-free Transformer architectures