Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs with Pollara interconnect. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts on Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.
6923c13eb5612535ed95587e
Quentin Anthony
6923c13eb5612535ed95587f
Yury Tokpanov
6923c13eb5612535ed955880
Skyler Szot
6923c13eb5612535ed955881
Srivatsan Rajagopal
6923c13eb5612535ed955882
Praneeth Medepalli
6923c13eb5612535ed955883
Rishi Iyer
6923c13eb5612535ed955884
Vasu Shyam
6923c13eb5612535ed955885
Anna Golubeva
6923c13eb5612535ed955886
Ansh Chaurasia
6923c13eb5612535ed955887
Xiao Yang
6923c13eb5612535ed955888
Tomas Figliolia
6923c13eb5612535ed955889
Robert Washbourne
claimed_verified
2025-11-25T09:06:03.012Z
6322542d6b1992383fa6d232
https://cdn-avatars.huggingface.co/v1/production/uploads/6322542d6b1992383fa6d232/jwJifrumSFVVKOLYm1LVa.jpeg
robert washbourne
rawsh
user
rawsh
6923c13eb5612535ed95588a
Drew Thorstensen
6923c13eb5612535ed95588b
Amartey Pearson
6923c13eb5612535ed95588c
Zack Grossbart
6923c13eb5612535ed95588d
Jason van Patten
6923c13eb5612535ed95588e
Emad Barsoum
6923c13eb5612535ed95588f
Zhenyu Gu
6923c13eb5612535ed955890
Yao Fu
6923c13eb5612535ed955891
Beren Millidge
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs with Pollara interconnect. We distill practical guidance for both systems and model design. On the systems side, we deliver a comprehensive cluster and networking characterization: microbenchmarks for all core collectives (all-reduce, reduce-scatter, all-gather, broadcast) across message sizes and GPU counts on Pollara. To our knowledge, this is the first at this scale. We further provide MI300X microbenchmarks on kernel sizing and memory bandwidth to inform model design. On the modeling side, we introduce and apply MI300X-aware transformer sizing rules for attention and MLP blocks and justify MoE widths that jointly optimize training throughput and inference latency. We describe our training stack in depth, including often-ignored utilities such as fault-tolerance and checkpoint-reshaping, as well as detailed information on our training recipe. We also provide a preview of our model architecture and base model - ZAYA1 (760M active, 8.3B total parameters MoE) - which will be further improved upon in forthcoming papers. ZAYA1-base achieves performance comparable to leading base models such as Qwen3-4B and Gemma3-12B at its scale and larger, and outperforms models including Llama-3-8B and OLMoE across reasoning, mathematics, and coding benchmarks. Together, these results demonstrate that the AMD hardware, network, and software stack are mature and optimized enough for competitive large-scale pretraining.
A large-scale mixture-of-experts (MoE) pretraining study on AMD hardware provides comprehensive guidance for system and model design, demonstrating competitive performance with leading models.
mixture-of-experts (MoE) MI300X GPUs Pollara interconnect all-reduce reduce-scatter all-gather broadcast kernel sizing memory bandwidth transformer sizing attention MLP blocks fault-tolerance checkpoint-reshaping ZAYA1 Qwen3-4B Gemma3-12B Llama-3-8B OLMoE