Scaling Laws for Code: Every Programming Language Matters
Code large language models (Code LLMs) are powerful but costly to train, with scaling laws predicting performance from model size, data, and compute. However, different programming languages (PLs) have varying impacts during pre-training that significantly affect base model performance, leading to inaccurate performance prediction. Besides, existing works focus on language-agnostic settings, neglecting the inherently multilingual nature of modern software development. Therefore, it is first necessary to investigate the scaling laws of different PLs, and then consider their mutual influences to arrive at the final multilingual scaling law. In this paper, we present the first systematic exploration of scaling laws for multilingual code pre-training, conducting over 1000+ experiments (Equivalent to 336,000+ H800 hours) across multiple PLs, model sizes (0.2B to 14B parameters), and dataset sizes (1T tokens). We establish comprehensive scaling laws for code LLMs across multiple PLs, revealing that interpreted languages (e.g., Python) benefit more from increased model size and data than compiled languages (e.g., Rust). The study demonstrates that multilingual pre-training provides synergistic benefits, particularly between syntactically similar PLs. Further, the pre-training strategy of the parallel pairing (concatenating code snippets with their translations) significantly enhances cross-lingual abilities with favorable scaling properties. Finally, a proportion-dependent multilingual scaling law is proposed to optimally allocate training tokens by prioritizing high-utility PLs (e.g., Python), balancing high-synergy pairs (e.g., JavaScript-TypeScript), and reducing allocation to fast-saturating languages (Rust), achieving superior average performance across all PLs compared to uniform distribution under the same compute budget.
694be9e1746a34b55dd53f86
Jian Yang
694be9e1746a34b55dd53f87
Shawn Guo
694be9e1746a34b55dd53f88
Lin Jing
694be9e1746a34b55dd53f89
Wei Zhang
claimed_verified
2026-03-27T12:58:04.762Z
69c67b2fa994b07915a6e083
/avatars/3d0fd966df540d34095d2c84ce449180.svg
wei zhang
zwpride
user
zwpride
694be9e1746a34b55dd53f8a
Aishan Liu
694be9e1746a34b55dd53f8b
Chuan Hao
694be9e1746a34b55dd53f8c
Zhoujun Li
694be9e1746a34b55dd53f8d
Wayne Xin Zhao
694be9e1746a34b55dd53f8e
Xianglong Liu
694be9e1746a34b55dd53f8f
Weifeng Lv
694be9e1746a34b55dd53f90
Bryan Dai
Scaling Laws for Code: Every Programming Language Matters
Code large language models (Code LLMs) are powerful but costly to train, with scaling laws predicting performance from model size, data, and compute. However, different programming languages (PLs) have varying impacts during pre-training that significantly affect base model performance, leading to inaccurate performance prediction. Besides, existing works focus on language-agnostic settings, neglecting the inherently multilingual nature of modern software development. Therefore, it is first necessary to investigate the scaling laws of different PLs, and then consider their mutual influences to arrive at the final multilingual scaling law. In this paper, we present the first systematic exploration of scaling laws for multilingual code pre-training, conducting over 1000+ experiments (Equivalent to 336,000+ H800 hours) across multiple PLs, model sizes (0.2B to 14B parameters), and dataset sizes (1T tokens). We establish comprehensive scaling laws for code LLMs across multiple PLs, revealing that interpreted languages (e.g., Python) benefit more from increased model size and data than compiled languages (e.g., Rust). The study demonstrates that multilingual pre-training provides synergistic benefits, particularly between syntactically similar PLs. Further, the pre-training strategy of the parallel pairing (concatenating code snippets with their translations) significantly enhances cross-lingual abilities with favorable scaling properties. Finally, a proportion-dependent multilingual scaling law is proposed to optimally allocate training tokens by prioritizing high-utility PLs (e.g., Python), balancing high-synergy pairs (e.g., JavaScript-TypeScript), and reducing allocation to fast-saturating languages (Rust), achieving superior average performance across all PLs compared to uniform distribution under the same compute budget.
Systematic exploration of scaling laws for multilingual code pre-training reveals language-specific benefits and proposes a strategy for optimal token allocation across programming languages.
Code LLMs programming languages scaling laws pre-training multilingual pre-training interpreted languages compiled languages parallel pairing proportion-dependent multilingual scaling law