Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive compute power and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. In this paper, we introduce Soup Of Category Experts (SoCE), a principled approach for model souping that utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous uniform-averaging approaches, our method leverages the observation that benchmark categories often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that the proposed method improves performance and robustness across multiple domains, including multilingual capabilities, tool calling, and math and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.
691c1c836bfd5965c0fd39e4
Shalini Maiti
claimed_verified
2025-11-18T14:06:47.486Z
6419b34a9a27800807c34a63
/avatars/ee1391a9a153bae0dd04323b1fa5b5d6.svg
Shalini M
shalinimaiti
user
shalinimaiti
691c1c836bfd5965c0fd39e5
Amar Budhiraja
claimed_verified
2025-11-18T14:06:49.892Z
6687ee79eee600e418404cc9
/avatars/d73d3a360ab3b3ce353a6306c7270a13.svg
Amar Budhiraja
ambud26
user
ambud26
691c1c836bfd5965c0fd39e6
Bhavul Gauri
claimed_verified
2025-11-19T08:40:31.124Z
60720704227ff331937110f4
/avatars/8010bfb98256c138049aa3d237737b37.svg
Bhavul Gauri
bhavul
user
bhavul
691c1c836bfd5965c0fd39e7
Gaurav Chaurasia
claimed_verified
2025-11-18T14:06:45.558Z
691c6b9b660c15d270b5838a
/avatars/2dc40e079bd8b7af7ae4a4ac43acc552.svg
Gaurav Chaurasia
gchauras
user
gchauras
691c1c836bfd5965c0fd39e8
Anton Protopopov
claimed_verified
2026-02-12T20:29:10.089Z
67f00bf17530c3fccbb26c79
/avatars/f0d56f04b1def33dce872a8de71f560d.svg
Anton Protopopov
aprotopopov
user
aprotopopov
691c1c836bfd5965c0fd39e9
Alexis Audran-Reiss
691c1c836bfd5965c0fd39ea
Michael Slater
691c1c836bfd5965c0fd39eb
Despoina Magka
claimed_verified
2026-04-13T19:26:51.899Z
673dc2345b07d224bff10173
/avatars/2b7b1491de77bc3193fd912de76e176d.svg
Despoina Magka
marlamagka
user
marlamagka
691c1c836bfd5965c0fd39ec
Tatiana Shavrina
691c1c836bfd5965c0fd39ed
Roberta Raileanu
691c1c836bfd5965c0fd39ee
Yoram Bachrach
Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive compute power and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. In this paper, we introduce Soup Of Category Experts (SoCE), a principled approach for model souping that utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous uniform-averaging approaches, our method leverages the observation that benchmark categories often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that the proposed method improves performance and robustness across multiple domains, including multilingual capabilities, tool calling, and math and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.