Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval
Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.
69d4e70794acf51b89ca46bc
Youngjoon Jang
claimed_verified
2026-04-08T08:06:21.120Z
65a4c4ed2548c41ad9b1421c
https://cdn-avatars.huggingface.co/v1/production/uploads/65a4c4ed2548c41ad9b1421c/bMQbowjHKvq-bKpzalvWm.jpeg
Youngjoon Jang
yjoonjang
user
yjoonjang
69d4e70794acf51b89ca46bd
Seongtae Hong
claimed_verified
2026-04-13T19:24:45.136Z
6541e7afe239b5e8d5060a0a
/avatars/07afe0201add8400e817ebf489d015ef.svg
Seongtae Hong
hongst
user
hongst
69d4e70794acf51b89ca46be
Hyeonseok Moon
69d4e70794acf51b89ca46bf
Heuiseok Lim
Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval
Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.
Stratified sampling improves knowledge distillation by preserving the full range of teacher scores, outperforming traditional sampling methods in retrieval tasks.
Knowledge Distillation cross-encoder hard negatives teacher score distribution stratified sampling retrieval models distillation