veScale-FSDP: Flexible and High-Performance FSDP at Scale
Fully Sharded Data Parallel (FSDP), also known as ZeRO, is widely used for training large-scale models, featuring its flexibility and minimal intrusion on model code. However, current FSDP systems struggle with structure-aware training methods (e.g., block-wise quantized training) and with non-element-wise optimizers (e.g., Shampoo and Muon) used in cutting-edge models (e.g., Gemini, Kimi K2). FSDP's fixed element- or row-wise sharding formats conflict with the block-structured computations. In addition, today's implementations fall short in communication and memory efficiency, limiting scaling to tens of thousands of GPUs. We introduce veScale-FSDP, a redesigned FSDP system that couples a flexible sharding format, RaggedShard, with a structure-aware planning algorithm to deliver both flexibility and performance at scale. veScale-FSDP natively supports efficient data placement required by FSDP, empowering block-wise quantization and non-element-wise optimizers. As a result, veScale-FSDP achieves 5~66% higher throughput and 16~30% lower memory usage than existing FSDP systems, while scaling efficiently to tens of thousands of GPUs.
69a108daa13deaa44944894c
Zezhou Wang
69a108daa13deaa44944894d
Youjie Li
69a108daa13deaa44944894e
Zhiqi Lin
69a108daa13deaa44944894f
Jiacheng Yang
69a108daa13deaa449448950
Cong Xie
69a108daa13deaa449448951
Guanyu Feng
69a108daa13deaa449448952
Zheng Zhong
69a108daa13deaa449448953
Ziyue Huang
69a108daa13deaa449448954
Hongyu Zhu
69a108daa13deaa449448955
Zhi Zhang
69a108daa13deaa449448956
Yanghua Peng
69a108daa13deaa449448957
Xin Liu
veScale-FSDP: Flexible and High-Performance FSDP at Scale
Fully Sharded Data Parallel (FSDP), also known as ZeRO, is widely used for training large-scale models, featuring its flexibility and minimal intrusion on model code. However, current FSDP systems struggle with structure-aware training methods (e.g., block-wise quantized training) and with non-element-wise optimizers (e.g., Shampoo and Muon) used in cutting-edge models (e.g., Gemini, Kimi K2). FSDP's fixed element- or row-wise sharding formats conflict with the block-structured computations. In addition, today's implementations fall short in communication and memory efficiency, limiting scaling to tens of thousands of GPUs. We introduce veScale-FSDP, a redesigned FSDP system that couples a flexible sharding format, RaggedShard, with a structure-aware planning algorithm to deliver both flexibility and performance at scale. veScale-FSDP natively supports efficient data placement required by FSDP, empowering block-wise quantization and non-element-wise optimizers. As a result, veScale-FSDP achieves 5~66% higher throughput and 16~30% lower memory usage than existing FSDP systems, while scaling efficiently to tens of thousands of GPUs.
veScale-FSDP introduces a redesigned fully sharded data parallel system with flexible sharding and structure-aware planning to improve scalability and efficiency for large-scale model training.
Fully Sharded Data Parallel ZeRO block-wise quantized training non-element-wise optimizers Shampoo Muon RaggedShard structure-aware planning communication efficiency memory efficiency scalability