UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
In recent years, the scaling laws of recommendation models have attracted increasing attention, which govern the relationship between performance and parameters/FLOPs of recommenders. Currently, there are three mainstream architectures for achieving scaling in recommendation models, namely attention-based, TokenMixer-based, and factorization-machine-based methods, which exhibit fundamental differences in both design philosophy and architectural structure. In this paper, we propose a unified scaling architecture for recommendation systems, namely UniMixer, to improve scaling efficiency and establish a unified theoretical framework that unifies the mainstream scaling blocks. By transforming the rule-based TokenMixer to an equivalent parameterized structure, we construct a generalized parameterized feature mixing module that allows the token mixing patterns to be optimized and learned during model training. Meanwhile, the generalized parameterized token mixing removes the constraint in TokenMixer that requires the number of heads to be equal to the number of tokens. Furthermore, we establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods. To further boost scaling ROI, a lightweight UniMixing module is designed, UniMixing-Lite, which further compresses the model parameters and computational cost while significantly improve the model performance. The scaling curves are shown in the following figure. Extensive offline and online experiments are conducted to verify the superior scaling abilities of UniMixer.
69cdfc0aacf91530b21dfa51
Mingming Ha
69cdfc0aacf91530b21dfa52
Guanchen Wang
69cdfc0aacf91530b21dfa53
Linxun Chen
69cdfc0aacf91530b21dfa54
Xuan Rao
69cdfc0aacf91530b21dfa55
Yuexin Shi
69cdfc0aacf91530b21dfa56
Tianbao Ma
69cdfc0aacf91530b21dfa57
Zhaojie Liu
69cdfc0aacf91530b21dfa58
Yunqian Fan
69cdfc0aacf91530b21dfa59
Zilong Lu
69cdfc0aacf91530b21dfa5a
Yanan Niu
69cdfc0aacf91530b21dfa5b
Han Li
69cdfc0aacf91530b21dfa5c
Kun Gai
UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
In recent years, the scaling laws of recommendation models have attracted increasing attention, which govern the relationship between performance and parameters/FLOPs of recommenders. Currently, there are three mainstream architectures for achieving scaling in recommendation models, namely attention-based, TokenMixer-based, and factorization-machine-based methods, which exhibit fundamental differences in both design philosophy and architectural structure. In this paper, we propose a unified scaling architecture for recommendation systems, namely UniMixer, to improve scaling efficiency and establish a unified theoretical framework that unifies the mainstream scaling blocks. By transforming the rule-based TokenMixer to an equivalent parameterized structure, we construct a generalized parameterized feature mixing module that allows the token mixing patterns to be optimized and learned during model training. Meanwhile, the generalized parameterized token mixing removes the constraint in TokenMixer that requires the number of heads to be equal to the number of tokens. Furthermore, we establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods. To further boost scaling ROI, a lightweight UniMixing module is designed, UniMixing-Lite, which further compresses the model parameters and computational cost while significantly improve the model performance. The scaling curves are shown in the following figure. Extensive offline and online experiments are conducted to verify the superior scaling abilities of UniMixer.
A unified scaling architecture for recommendation systems called UniMixer is proposed, featuring a generalized parameterized token mixing module that optimizes token mixing patterns and improves scaling efficiency across different architectures.
scaling laws recommendation models attention-based TokenMixer-based factorization-machine-based UniMixer parameterized structure feature mixing module token mixing patterns UniMixing-Lite scaling ROI