Motif 2 12.7B technical report
We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models.
691623a021e02bff6aac7a95
Junghwan Lim
691623a021e02bff6aac7a96
Sungmin Lee
691623a021e02bff6aac7a97
Dongseok Kim
691623a021e02bff6aac7a98
Taehyun Kim
691623a021e02bff6aac7a99
Eunhwan Park
691623a021e02bff6aac7a9a
Jeesoo Lee
691623a021e02bff6aac7a9b
Jeongdoo Lee
691623a021e02bff6aac7a9c
Junhyeok Lee
claimed_verified
2025-11-24T07:59:20.065Z
64c0c845a04a514ba62bcd1a
/avatars/6e975f14e196f1fa882f8c52921f8f46.svg
leejunhyeok
leejunhyeok
user
leejunhyeok
691623a021e02bff6aac7a9d
Wai Ting Cheung
claimed_verified
2025-11-17T10:31:46.934Z
654063b699e887c04054a88c
/avatars/a4760a9d008ef57efa01008ca406c658.svg
Ken Cheung
kencwt
user
kencwt
691623a021e02bff6aac7a9e
Dahye Choi
691623a021e02bff6aac7a9f
Jaeheui Her
691623a021e02bff6aac7aa0
Jaeyeon Huh
691623a021e02bff6aac7aa1
Hanbin Jung
691623a021e02bff6aac7aa2
Changjin Kang
691623a021e02bff6aac7aa3
Beomgyu Kim
691623a021e02bff6aac7aa4
Minjae Kim
691623a021e02bff6aac7aa5
Taewhan Kim
691623a021e02bff6aac7aa6
Youngrok Kim
691623a021e02bff6aac7aa7
Hyukjin Kweon
691623a021e02bff6aac7aa8
Haesol Lee
691623a021e02bff6aac7aa9
Kungyu Lee
691623a021e02bff6aac7aaa
Dongpin Oh
691623a021e02bff6aac7aab
Yeongjae Park
691623a021e02bff6aac7aac
Bokki Ryu
691623a021e02bff6aac7aad
Dongjoo Weon
Motif 2 12.7B technical report
We introduce Motif-2-12.7B, a new open-weight foundation model that pushes the efficiency frontier of large language models by combining architectural innovation with system-level optimization. Designed for scalable language understanding and robust instruction generalization under constrained compute budgets, Motif-2-12.7B builds upon Motif-2.6B with the integration of Grouped Differential Attention (GDA), which improves representational efficiency by disentangling signal and noise-control attention pathways. The model is pre-trained on 5.5 trillion tokens spanning diverse linguistic, mathematical, scientific, and programming domains using a curriculum-driven data scheduler that gradually changes the data composition ratio. The training system leverages the MuonClip optimizer alongside custom high-performance kernels, including fused PolyNorm activations and the Parallel Muon algorithm, yielding significant throughput and memory efficiency gains in large-scale distributed environments. Post-training employs a three-stage supervised fine-tuning pipeline that successively enhances general instruction adherence, compositional understanding, and linguistic precision. Motif-2-12.7B demonstrates competitive performance across diverse benchmarks, showing that thoughtful architectural scaling and optimized training design can rival the capabilities of much larger models.
Motif-2-12.7B combines architectural innovations and system optimizations to enhance efficiency and performance in large language models.
Grouped Differential Attention (GDA) PolyNorm activations Parallel Muon algorithm curriculum-driven data scheduler MuonClip optimizer fused PolyNorm activations