Instella: Fully Open Language Models with Stellar Performance
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, yet the majority of high-performing models remain closed-source or partially open, limiting transparency and reproducibility. In this work, we introduce Instella, a family of fully open three billion parameter language models trained entirely on openly available data and codebase. Powered by AMD Instinct MI300X GPUs, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences. Despite using substantially fewer pre-training tokens than many contemporaries, Instella achieves state-of-the-art results among fully open models and is competitive with leading open-weight models of comparable size. We further release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning on mathematical tasks. Together, these contributions establish Instella as a transparent, performant, and versatile alternative for the community, advancing the goal of open and reproducible language modeling research.
6916f8f93b377710cdf19a4b
Jiang Liu
6916f8f93b377710cdf19a4c
Jialian Wu
6916f8f93b377710cdf19a4d
Xiaodong Yu
6916f8f93b377710cdf19a4e
Yusheng Su
6916f8f93b377710cdf19a4f
Prakamya Mishra
6916f8f93b377710cdf19a50
Gowtham Ramesh
6916f8f93b377710cdf19a51
Sudhanshu Ranjan
6916f8f93b377710cdf19a52
Chaitanya Manem
6916f8f93b377710cdf19a53
Ximeng Sun
6916f8f93b377710cdf19a54
Ze Wang
6916f8f93b377710cdf19a55
Pratik Prabhanjan Brahma
6916f8f93b377710cdf19a56
Zicheng Liu
6916f8f93b377710cdf19a57
Emad Barsoum
Instella: Fully Open Language Models with Stellar Performance
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, yet the majority of high-performing models remain closed-source or partially open, limiting transparency and reproducibility. In this work, we introduce Instella, a family of fully open three billion parameter language models trained entirely on openly available data and codebase. Powered by AMD Instinct MI300X GPUs, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences. Despite using substantially fewer pre-training tokens than many contemporaries, Instella achieves state-of-the-art results among fully open models and is competitive with leading open-weight models of comparable size. We further release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning on mathematical tasks. Together, these contributions establish Instella as a transparent, performant, and versatile alternative for the community, advancing the goal of open and reproducible language modeling research.
Instella, a family of fully open large language models, achieves state-of-the-art performance using open data and is competitive with leading open-weight models, with specialized variants for long context and mathematical reasoning.
large language models AMD Instinct MI300X GPUs large-scale pre-training general-purpose instruction tuning alignment with human preferences Instella Instella-Long Instella-Math supervised fine-tuning reinforcement learning