Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers
The scaling of Large Language Models (LLMs) drives interest in matrix-based optimizers (e.g., Shampoo, Muon, SOAP) for their convergence efficiency; yet their requirement for holistic updates conflicts with the tensor fragmentation in distributed frameworks like Megatron. Existing solutions are suboptimal: synchronous approaches suffer from computational redundancy, while layer-wise partitioning fails to reconcile this conflict without violating the geometric constraints of efficient communication primitives. To bridge this gap, we propose Canzona, a Unified, Asynchronous, and Load-Balanced framework that decouples logical optimizer assignment from physical parameter distribution. For Data Parallelism, we introduce an alpha-Balanced Static Partitioning strategy that respects atomicity while neutralizing the load imbalance. For Tensor Parallelism, we design an Asynchronous Compute pipeline utilizing Micro-Group Scheduling to batch fragmented updates and hide reconstruction overhead. Extensive evaluations on the Qwen3 model family (up to 32B parameters) on 256 GPUs demonstrate that our approach preserves the efficiency of established parallel architectures, achieving a 1.57x speedup in end-to-end iteration time and reducing optimizer step latency by 5.8x compared to the baseline.
69895559beecc443208d26a2
Liangyu Wang
claimed_verified
2026-02-11T22:17:13.834Z
66224a84afbc88c1e4881ad7
https://cdn-avatars.huggingface.co/v1/production/uploads/66224a84afbc88c1e4881ad7/fGDiIiqhTBQri3khSqNcU.jpeg
Liangyu Wang
ly4096
user
ly4096
69895559beecc443208d26a3
Siqi Zhang
69895559beecc443208d26a4
Junjie Wang
69895559beecc443208d26a5
Yiming Dong
69895559beecc443208d26a6
Bo Zheng
69895559beecc443208d26a7
Zihan Qiu
69895559beecc443208d26a8
Shengkun Tang
69895559beecc443208d26a9
Di Wang
69895559beecc443208d26aa
Rui Men
69895559beecc443208d26ab
Dayiheng Liu
Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers
The scaling of Large Language Models (LLMs) drives interest in matrix-based optimizers (e.g., Shampoo, Muon, SOAP) for their convergence efficiency; yet their requirement for holistic updates conflicts with the tensor fragmentation in distributed frameworks like Megatron. Existing solutions are suboptimal: synchronous approaches suffer from computational redundancy, while layer-wise partitioning fails to reconcile this conflict without violating the geometric constraints of efficient communication primitives. To bridge this gap, we propose Canzona, a Unified, Asynchronous, and Load-Balanced framework that decouples logical optimizer assignment from physical parameter distribution. For Data Parallelism, we introduce an alpha-Balanced Static Partitioning strategy that respects atomicity while neutralizing the load imbalance. For Tensor Parallelism, we design an Asynchronous Compute pipeline utilizing Micro-Group Scheduling to batch fragmented updates and hide reconstruction overhead. Extensive evaluations on the Qwen3 model family (up to 32B parameters) on 256 GPUs demonstrate that our approach preserves the efficiency of established parallel architectures, achieving a 1.57x speedup in end-to-end iteration time and reducing optimizer step latency by 5.8x compared to the baseline.
Canzona presents a unified asynchronous framework that addresses the conflict between matrix-based optimizers and distributed tensor fragmentation in LLM training, improving efficiency and reducing latency.
Large Language Models matrix-based optimizers Shampoo Muon SOAP distributed frameworks Megatron synchronous approaches layer-wise partitioning geometric constraints Canzona Data Parallelism alpha-Balanced Static Partitioning tensor fragmentation Tensor Parallelism Asynchronous Compute pipeline Micro-Group Scheduling optimizer step latency end-to-end iteration time