Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.
text
Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.
new_line
new_line
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
text
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
new_line
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
text
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
new_line
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
text
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
new_line
• Move gradient division to the very end of the backward pass.
text
• Move gradient division to the very end of the backward pass.
new_line
new_line
To learn more about how YaFSDP works, check out our latest blog post: 
text
To learn more about how YaFSDP works, check out our latest blog post: 
https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
link
Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.

Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.

To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
👍
Omshirsat kristaller486 itsdaria GPT007 umutphp Joseph717171 MaziyarPanahi den0620 Rexhaif ucyang irvingjr Ol9 ermakovpetr
❤️
Joseph717171 MaziyarPanahi Rexhaif Ol9 ermakovpetr
🔥
Joseph717171 MaziyarPanahi Rexhaif Ol9 ermakovpetr
🚀
Joseph717171 Ol9 ermakovpetr
🤗
Joseph717171 Ol9 ermakovpetr
🤝
Joseph717171 Ol9 ermakovpetr