Arcee Trinity Large Technical Report
We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.
6997cd777a658569d5a10149
Varun Singh
6997cd777a658569d5a1014a
Lucas Krauss
6997cd777a658569d5a1014b
Sami Jaghouar
6997cd777a658569d5a1014c
Matej Sirovatka
admin_assigned
2026-02-20T09:07:12.184Z
658f19cfa02954c982f540eb
https://cdn-avatars.huggingface.co/v1/production/uploads/658f19cfa02954c982f540eb/qN2eqJZnWQ2H1xlGYeAKj.jpeg
Matej Sirovatka
siro1
user
siro1
6997cd777a658569d5a1014d
Charles Goddard
admin_assigned
2026-02-20T09:07:18.720Z
630495b0ce6b12280b193c25
https://cdn-avatars.huggingface.co/v1/production/uploads/630495b0ce6b12280b193c25/bT61kBtQPhDYk00AI0o0g.png
Charles Goddard
chargoddard
user
chargoddard
6997cd777a658569d5a1014e
Fares Obied
6997cd777a658569d5a1014f
Jack Min Ong
admin_assigned
2026-02-20T09:07:29.527Z
634e2b60a00c472888747e4c
https://cdn-avatars.huggingface.co/v1/production/uploads/634e2b60a00c472888747e4c/FGqQ1qcfN0pXZ9GgdRrBB.png
Jack Min Ong
Jackmin108
user
Jackmin108
6997cd777a658569d5a10150
Jannik Straube
6997cd777a658569d5a10151
Fern
6997cd777a658569d5a10152
Aria Harley
6997cd777a658569d5a10153
Conner Stewart
admin_assigned
2026-02-20T09:07:40.441Z
67f8381b383e0a7c03504e01
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/De8SEV0r_koGe4Ke7KBgD.png
Conner Stewart
Chemicon0302
user
Chemicon0302
6997cd777a658569d5a10154
Colin Kealty
6997cd777a658569d5a10155
Maziyar Panahi
admin_assigned
2026-02-20T09:07:50.837Z
5fd5e18a90b6dc4633f6d292
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd5e18a90b6dc4633f6d292/gZXHW5dd9R86AV9LMZ--y.png
Maziyar Panahi
MaziyarPanahi
user
MaziyarPanahi
6997cd777a658569d5a10156
Simon Kirsten
6997cd777a658569d5a10157
Anushka Deshpande
6997cd777a658569d5a10158
Anneketh Vij
claimed_verified
2026-02-23T09:46:09.321Z
664aa483babc0b3356a35cc3
/avatars/e915679eb88d05faebaad28957955c9b.svg
Anneketh Vij
annekethvij
user
annekethvij
6997cd777a658569d5a10159
Arthur Bresnu
admin_assigned
2026-02-20T09:08:23.592Z
663ce8bfc6827e9ba288eed4
https://cdn-avatars.huggingface.co/v1/production/uploads/663ce8bfc6827e9ba288eed4/0xrGyvLOq8DtP4TQsesnS.jpeg
Arthur BRESNU
arthurbresnu
user
arthurbresnu
6997cd777a658569d5a1015a
Pranav Veldurthi
admin_assigned
2026-02-20T09:08:30.013Z
64b6c10161dc301f13235b47
/avatars/9f4f434e855077a33698560eaff0c58e.svg
Pranav Veldurthi
pranavkr
user
pranavkr
6997cd777a658569d5a1015b
Raghav Ravishankar
admin_assigned
2026-02-20T09:08:36.828Z
6434d35aa5aed21dd11a294d
/avatars/4af84f8850c0386e706b5537b5c48b26.svg
Raghav Ravishankar
Alyosha11
user
Alyosha11
6997cd777a658569d5a1015c
Hardik Bishnoi
6997cd777a658569d5a1015d
DatologyAI Team
6997cd777a658569d5a1015e
Arcee AI Team
6997cd777a658569d5a1015f
Prime Intellect Team
6997cd777a658569d5a10160
Mark McQuade
6997cd777a658569d5a10161
Johannes Hagemann
6997cd777a658569d5a10162
Lucas Atkins
Arcee Trinity Large Technical Report
We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.
Arcee Trinity models are sparse Mixture-of-Experts architectures with varying parameter counts and activation patterns, utilizing advanced attention mechanisms and training optimizations.
Mixture-of-Experts sparse Mixture-of-Experts attention gated attention depth-scaled sandwich norm sigmoid routing Muon optimizer MoE load balancing Soft-clamped Momentum Expert Bias Updates