Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation
Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are crucial for guiding research and optimizing resource allocation. We hypothesize that this may be attributed to the inherent noise, bias, and incompleteness of raw user interaction data in prior continual pre-training (CPT) efforts. This paper introduces a novel, layered framework for generating high-quality synthetic data that circumvents such issues by creating a curated, pedagogical curriculum for the LLM. We provide powerful, direct evidence for the utility of our curriculum by showing that standard sequential models trained on our principled synthetic data significantly outperform (+130% on recall@100 for SasRec) models trained on real data in downstream ranking tasks, demonstrating its superiority for learning generalizable user preference patterns. Building on this, we empirically demonstrate, for the first time, robust power-law scaling for an LLM that is continually pre-trained on our high-quality, recommendation-specific data. Our experiments reveal consistent and predictable perplexity reduction across multiple synthetic data modalities. These findings establish a foundational methodology for reliable scaling LLM capabilities in the recommendation domain, thereby shifting the research focus from mitigating data deficiencies to leveraging high-quality, structured information.
69939e5f50fb2c0be4783c7f
Benyu Zhang
69939e5f50fb2c0be4783c80
Qiang Zhang
69939e5f50fb2c0be4783c81
Jianpeng Cheng
69939e5f50fb2c0be4783c82
Hong-You Chen
69939e5f50fb2c0be4783c83
Qifei Wang
69939e5f50fb2c0be4783c84
Wei Sun
69939e5f50fb2c0be4783c85
Shen Li
69939e5f50fb2c0be4783c86
Jia Li
69939e5f50fb2c0be4783c87
Jiahao Wu
69939e5f50fb2c0be4783c88
Xiangjun Fan
69939e5f50fb2c0be4783c89
Hong Yan
Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation
Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are crucial for guiding research and optimizing resource allocation. We hypothesize that this may be attributed to the inherent noise, bias, and incompleteness of raw user interaction data in prior continual pre-training (CPT) efforts. This paper introduces a novel, layered framework for generating high-quality synthetic data that circumvents such issues by creating a curated, pedagogical curriculum for the LLM. We provide powerful, direct evidence for the utility of our curriculum by showing that standard sequential models trained on our principled synthetic data significantly outperform (+130% on recall@100 for SasRec) models trained on real data in downstream ranking tasks, demonstrating its superiority for learning generalizable user preference patterns. Building on this, we empirically demonstrate, for the first time, robust power-law scaling for an LLM that is continually pre-trained on our high-quality, recommendation-specific data. Our experiments reveal consistent and predictable perplexity reduction across multiple synthetic data modalities. These findings establish a foundational methodology for reliable scaling LLM capabilities in the recommendation domain, thereby shifting the research focus from mitigating data deficiencies to leveraging high-quality, structured information.
A novel layered framework generates high-quality synthetic data for large language models in recommender systems, demonstrating superior performance and predictable scaling laws compared to traditional methods.
large language models recommender systems continual pre-training synthetic data sequential models recall@100 power-law scaling perplexity user interaction data pedagogical curriculum