Revisiting Parameter Server in LLM Post-Training
Modern data parallel (DP) training favors collective communication over parameter servers (PS) for its simplicity and efficiency under balanced workloads. However, the balanced workload assumption no longer holds in large language model (LLM) post-training due to the high variance in sequence lengths. Under imbalanced workloads, collective communication creates synchronization barriers, leading to under-utilization of devices with smaller workloads. This change in training dynamics calls for a revisit of the PS paradigm for its robustness to such imbalance. We propose On-Demand Communication (ODC), which adapts PS into Fully Sharded Data Parallel (FSDP) by replacing collective all-gather and reduce-scatter with direct point-to-point communication. Compared to FSDP, ODC reduces the synchronization barrier from once per layer to once per minibatch and decouples the workload on each device so that faster workers are not stalled. It also enables simpler and more effective load balancing at the minibatch level. Across diverse LLM post-training tasks, ODC consistently improves device utilization and training throughput, achieving up to a 36\% speedup over standard FSDP. These results demonstrate that ODC is a superior fit for the prevalent imbalanced workloads in LLM post-training. Our implementation of ODC and integration with FSDP is open-sourced at https://github.com/sail-sg/odc.
69797926df44b75fa47e4748
Xinyi Wan
claimed_verified
2026-01-28T11:16:39.482Z
63510eea0b94548566dad923
/avatars/629eaaf810718259bf7588dc2e6cc0d5.svg
Xinyi Wan
ufotalent
user
ufotalent
69797926df44b75fa47e4749
Penghui Qi
claimed_verified
2026-02-02T17:00:51.366Z
63885f1d0bebb233d8ad6e5b
https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
Penghui Qi
QPHutu
user
QPHutu
69797926df44b75fa47e474a
Guangxing Huang
69797926df44b75fa47e474b
Chaoyi Ruan
69797926df44b75fa47e474c
Min Lin
69797926df44b75fa47e474d
Jialin Li
Revisiting Parameter Server in LLM Post-Training
Modern data parallel (DP) training favors collective communication over parameter servers (PS) for its simplicity and efficiency under balanced workloads. However, the balanced workload assumption no longer holds in large language model (LLM) post-training due to the high variance in sequence lengths. Under imbalanced workloads, collective communication creates synchronization barriers, leading to under-utilization of devices with smaller workloads. This change in training dynamics calls for a revisit of the PS paradigm for its robustness to such imbalance. We propose On-Demand Communication (ODC), which adapts PS into Fully Sharded Data Parallel (FSDP) by replacing collective all-gather and reduce-scatter with direct point-to-point communication. Compared to FSDP, ODC reduces the synchronization barrier from once per layer to once per minibatch and decouples the workload on each device so that faster workers are not stalled. It also enables simpler and more effective load balancing at the minibatch level. Across diverse LLM post-training tasks, ODC consistently improves device utilization and training throughput, achieving up to a 36\% speedup over standard FSDP. These results demonstrate that ODC is a superior fit for the prevalent imbalanced workloads in LLM post-training. Our implementation of ODC and integration with FSDP is open-sourced at https://github.com/sail-sg/odc.
On-Demand Communication (ODC) adapts parameter server principles to Fully Sharded Data Parallel training by replacing collective communication with point-to-point communication, improving device utilization and throughput in imbalanced large language model training scenarios.
data parallel parameter servers collective communication Fully Sharded Data Parallel On-Demand Communication all-gather reduce-scatter point-to-point communication load balancing device utilization training throughput