Order Matters in the Presence of Dataset Imbalance for Multilingual
  Learning
In this paper, we empirically study the optimization dynamics of multi-task
learning, particularly focusing on those that govern a collection of tasks with
significant data imbalance. We present a simple yet effective method of
pre-training on high-resource tasks, followed by fine-tuning on a mixture of
high/low-resource tasks. We provide a thorough empirical study and analysis of
this method's benefits showing that it achieves consistent improvements
relative to the performance trade-off profile of standard static weighting. We
analyze under what data regimes this method is applicable and show its
improvements empirically in neural machine translation (NMT) and multi-lingual
language modeling.
6577efc7dd2996f01ae7077b
Dami Choi
admin_assigned
2023-12-12T08:58:53.917Z
641cacf592cd2530299819aa
/avatars/7e0ee9d19f05deb320f359b9625a0526.svg
Dami Choi
choidami
user
choidami
6577efc7dd2996f01ae7077c
Derrick Xin
admin_assigned
2023-12-12T08:59:00.772Z
634cea1605f736dff37abe84
/avatars/6c199de4b6e7fd1320f47eeecd1310f2.svg
Derrick Xin
Derr
user
Derr
6577efc7dd2996f01ae7077d
Hamid Dadkhahi
6577efc7dd2996f01ae7077e
Justin Gilmer
6577efc7dd2996f01ae7077f
Ankush Garg
admin_assigned
2023-12-12T08:59:12.353Z
609bfde2138207cc2f2f3a8c
/avatars/8bb0198645bf8aa4e473bab3b1d76b20.svg
garg
ankushgarg
user
ankushgarg
6577efc7dd2996f01ae70780
Orhan Firat
6577efc7dd2996f01ae70781
Chih-Kuan Yeh
6577efc7dd2996f01ae70782
Andrew M. Dai
admin_assigned
2023-12-12T08:59:29.785Z
64c04c0b5e03515457ed72a3
/avatars/dcbfca63b32a8d5c575232316c01dbe7.svg
Andrew Dai
PenguinPush
user
PenguinPush
6577efc7dd2996f01ae70783
Behrooz Ghorbani
Order Matters in the Presence of Dataset Imbalance for Multilingual
  Learning
In this paper, we empirically study the optimization dynamics of multi-task
learning, particularly focusing on those that govern a collection of tasks with
significant data imbalance. We present a simple yet effective method of
pre-training on high-resource tasks, followed by fine-tuning on a mixture of
high/low-resource tasks. We provide a thorough empirical study and analysis of
this method's benefits showing that it achieves consistent improvements
relative to the performance trade-off profile of standard static weighting. We
analyze under what data regimes this method is applicable and show its
improvements empirically in neural machine translation (NMT) and multi-lingual
language modeling.
A method of pre-training on high-resource tasks followed by fine-tuning on a mix of high/low-resource tasks improves performance in multi-task learning with data imbalance, particularly in neural machine translation and multi-lingual language modeling.
multi-task learning data imbalance pre-training fine-tuning static weighting neural machine translation multi-lingual language modeling