mSFT: Addressing Dataset Mixtures Overfiting Heterogeneously in Multi-task SFT
Current language model training commonly applies multi-task Supervised Fine-Tuning (SFT) using a homogeneous compute budget across all sub-datasets. This approach is fundamentally sub-optimal: heterogeneous learning dynamics cause faster-learning tasks to overfit early while slower ones remain under-fitted. To address this, we introduce mSFT, an iterative, overfitting-aware search algorithm for multi-task data mixtures. mSFT trains the model on an active mixture, identifies and excludes the earliest overfitting sub-dataset, and reverts to that specific optimal checkpoint before continuing. Extensive evaluations demonstrate that mSFT consistently outperforms 4 baselines across 10 benchmarks and 6 base models. Further analysis confirms mSFT maintains robust gains across diverse dataset sizes, task granularities, and is insensitive to its single new hyperparameter (compute budget). Notably, at low compute budget, mSFT can improve performance while lowering training FLOPs. Ultimately, mSFT establishes a practical overfitting-aware algorithm for multi-task SFT that maximizes the potential of models across diverse data mixtures.
69c1f5b7c105f45cb0481ddf
Woosung Koh
69c1f5b7c105f45cb0481de0
Jeyoung Jeon
claimed_verified
2026-03-24T09:22:07.057Z
67f69d2ab591fb468d811eab
/avatars/42a10999e6f2f99924667ebfe26105e1.svg
Jeyoung Jeon
jyngjeon
user
jyngjeon
69c1f5b7c105f45cb0481de1
Youngjin Song
69c1f5b7c105f45cb0481de2
Yujin Cheon
69c1f5b7c105f45cb0481de3
Soowon Oh
69c1f5b7c105f45cb0481de4
Jaehyeong Choi
69c1f5b7c105f45cb0481de5
Se-Young Yun
mSFT: Addressing Dataset Mixtures Overfiting Heterogeneously in Multi-task SFT
Current language model training commonly applies multi-task Supervised Fine-Tuning (SFT) using a homogeneous compute budget across all sub-datasets. This approach is fundamentally sub-optimal: heterogeneous learning dynamics cause faster-learning tasks to overfit early while slower ones remain under-fitted. To address this, we introduce mSFT, an iterative, overfitting-aware search algorithm for multi-task data mixtures. mSFT trains the model on an active mixture, identifies and excludes the earliest overfitting sub-dataset, and reverts to that specific optimal checkpoint before continuing. Extensive evaluations demonstrate that mSFT consistently outperforms 4 baselines across 10 benchmarks and 6 base models. Further analysis confirms mSFT maintains robust gains across diverse dataset sizes, task granularities, and is insensitive to its single new hyperparameter (compute budget). Notably, at low compute budget, mSFT can improve performance while lowering training FLOPs. Ultimately, mSFT establishes a practical overfitting-aware algorithm for multi-task SFT that maximizes the potential of models across diverse data mixtures.
Multi-task supervised fine-tuning with heterogeneous learning dynamics benefits from an iterative overfitting-aware search algorithm that improves performance across diverse datasets and compute budgets.
multi-task Supervised Fine-Tuning SFT overfitting iterative search algorithm data mixtures optimal checkpoint training FLOPs