An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift
Preference tuning aligns pretrained language models to human judgments of quality, helpfulness, or safety by optimizing over explicit preference signals rather than likelihood alone. Prior work has shown that preference-tuning degrades performance and reduces helpfulness when evaluated outside the training domain. However, the extent to which adaptation strategies mitigate this domain shift remains unexplored. We address this challenge by conducting a comprehensive and systematic study of alignment generalization under domain shift. We compare five popular alignment objectives and various adaptation strategies from source to target, including target-domain supervised fine-tuning and pseudo-labeling, across summarization and question-answering helpfulness tasks. Our findings reveal systematic differences in generalization across alignment objectives under domain shift. We show that adaptation strategies based on pseudo-labeling can substantially reduce domain-shift degradation
6964d4daba573026e23e33fd
Constantinos Karouzos
claimed_verified
2026-01-13T08:24:40.062Z
63fe346e8b3c5087ff87776d
/avatars/d0a8ad1e2d4243f232ad7744ad780464.svg
Constantinos Karouzos
ckarouzos
user
ckarouzos
6964d4daba573026e23e33fe
Xingwei Tan
claimed_verified
2026-01-12T14:16:37.605Z
643d0a4d8a55b2bbf4f2a90e
/avatars/9534aaf81cbf12f015c6826b682fdb84.svg
Xingwei Tan
XingweiT
user
XingweiT
6964d4daba573026e23e33ff
Nikolaos Aletras
An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift
Preference tuning aligns pretrained language models to human judgments of quality, helpfulness, or safety by optimizing over explicit preference signals rather than likelihood alone. Prior work has shown that preference-tuning degrades performance and reduces helpfulness when evaluated outside the training domain. However, the extent to which adaptation strategies mitigate this domain shift remains unexplored. We address this challenge by conducting a comprehensive and systematic study of alignment generalization under domain shift. We compare five popular alignment objectives and various adaptation strategies from source to target, including target-domain supervised fine-tuning and pseudo-labeling, across summarization and question-answering helpfulness tasks. Our findings reveal systematic differences in generalization across alignment objectives under domain shift. We show that adaptation strategies based on pseudo-labeling can substantially reduce domain-shift degradation
Preference tuning of language models shows varying generalization capabilities under domain shift, with pseudo-labeling adaptation strategies effectively reducing performance degradation in summarization and question-answering tasks.
preference tuning language models domain shift alignment objectives adaptation strategies supervised fine-tuning pseudo-labeling summarization question-answering