daVinci-LLM:Towards the Science of Pretraining
The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it remains critically under-explored. This stems from a structural paradox: organizations with computational resources operate under commercial pressures that inhibit transparent disclosure, while academic institutions possess research freedom but lack pretraining-scale computational resources. daVinci-LLM occupies this unexplored intersection, combining industrial-scale resources with full research freedom to advance the science of pretraining. We adopt a fully-open paradigm that treats openness as scientific methodology, releasing complete data processing pipelines, full training processes, and systematic exploration results. Recognizing that the field lacks systematic methodology for data processing, we employ the Data Darwinism framework, a principled L0-L9 taxonomy from filtering to synthesis. We train a 3B-parameter model from random initialization across 8T tokens using a two-stage adaptive curriculum that progressively shifts from foundational capabilities to reasoning-intensive enhancement. Through 200+ controlled ablations, we establish that: processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling; different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts; compositional balance enables targeted intensification while preventing performance collapse; how evaluation protocol choices shape our understanding of pretraining progress. By releasing the complete exploration process, we enable the community to build upon our findings and systematic methodologies to form accumulative scientific knowledge in pretraining.
69cb59f222f9e9dce3fe2151
Yiwei Qin
69cb59f222f9e9dce3fe2152
Yixiu Liu
69cb59f222f9e9dce3fe2153
Tiantian Mi
69cb59f222f9e9dce3fe2154
Muhang Xie
claimed_verified
2026-03-31T09:23:57.611Z
67b7f988462862d18db625cc
/avatars/aba4a89691b927d3ca95f387f3291a1a.svg
谢牧航
Midoria7
user
Midoria7
69cb59f222f9e9dce3fe2155
Zhen Huang
69cb59f222f9e9dce3fe2156
Weiye Si
69cb59f222f9e9dce3fe2157
Pengrui Lu
69cb59f222f9e9dce3fe2158
Siyuan Feng
69cb59f222f9e9dce3fe2159
Xia Wu
69cb59f222f9e9dce3fe215a
Liming Liu
69cb59f222f9e9dce3fe215b
Ye Luo
69cb59f222f9e9dce3fe215c
Jinlong Hou
69cb59f222f9e9dce3fe215d
Qipeng Guo
69cb59f222f9e9dce3fe215e
Yu Qiao
69cb59f222f9e9dce3fe215f
Pengfei Liu
daVinci-LLM:Towards the Science of Pretraining
The foundational pretraining phase determines a model's capability ceiling, as post-training struggles to overcome capability foundations established during pretraining, yet it remains critically under-explored. This stems from a structural paradox: organizations with computational resources operate under commercial pressures that inhibit transparent disclosure, while academic institutions possess research freedom but lack pretraining-scale computational resources. daVinci-LLM occupies this unexplored intersection, combining industrial-scale resources with full research freedom to advance the science of pretraining. We adopt a fully-open paradigm that treats openness as scientific methodology, releasing complete data processing pipelines, full training processes, and systematic exploration results. Recognizing that the field lacks systematic methodology for data processing, we employ the Data Darwinism framework, a principled L0-L9 taxonomy from filtering to synthesis. We train a 3B-parameter model from random initialization across 8T tokens using a two-stage adaptive curriculum that progressively shifts from foundational capabilities to reasoning-intensive enhancement. Through 200+ controlled ablations, we establish that: processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling; different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts; compositional balance enables targeted intensification while preventing performance collapse; how evaluation protocol choices shape our understanding of pretraining progress. By releasing the complete exploration process, we enable the community to build upon our findings and systematic methodologies to form accumulative scientific knowledge in pretraining.
The study explores pretraining methodology through an open scientific approach, demonstrating that data processing depth and adaptive curriculum strategies significantly impact model capability development.
pretraining data processing pipelines training processes systematic exploration Data Darwinism framework L0-L9 taxonomy adaptive curriculum random initialization token scaling controlled ablations compositional balance evaluation protocols