On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each neuron, with the "winner" determined by its initial spectral magnitude and phase alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.
6997d62f7a658569d5a101b1
Jianliang He
claimed_verified
2026-02-20T08:32:54.929Z
6852f69c7b834665dc38c837
/avatars/bf9c5fc72300756e88319ec45c75f337.svg
Jianliang He
JLiangHe
user
JLiangHe
6997d62f7a658569d5a101b2
Leda Wang
6997d62f7a658569d5a101b3
Siyu Chen
6997d62f7a658569d5a101b4
Zhuoran Yang
On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each neuron, with the "winner" determined by its initial spectral magnitude and phase alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.
Two-layer neural networks solve modular addition by learning Fourier features through phase symmetry and frequency diversification, enabling robust computation via majority voting despite individual neuron noise.
two-layer neural networks modular addition Fourier features phase symmetry frequency diversification overparametrized indicator function majority-voting scheme random initialization lottery ticket mechanism gradient flow analysis phase coupling dynamics ODE comparison lemma grokking memorization generalization phases loss minimization weight decay