Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization because they better capture semantic information, are better at reasoning, and are more robust to paraphrasing. However, LLM judges show biases for length and order among others, and are vulnerable to various adversarial input prompts. While recent studies have looked into these biases, few have analyzed them at a more granular level in relation to a well-defined overlap metric. In this work we provide an LLM judge bias analysis as a function of overlap with human-written responses in the domain of summarization. We test 9 recent LLMs with parameter counts ranging from 1 billion to 12 billion, including variants of Gemma 3 and LLaMA 3. We find that LLM judges increasingly prefer summaries generated by other LLMs over those written by humans as the similarities (as measured by ROUGE and BLEU) between the judged summaries decrease, and this pattern extends to all but one model tested, and exists regardless of the models' own position biases. Additionally, we find that models struggle to judge even summaries with limited overlaps, suggesting that LLM-as-a-judge in the summary domain should rely on techniques beyond a simple comparison.
6994361550fb2c0be4783ea0
Jiangnan Fang
6994361550fb2c0be4783ea1
Cheng-Tse Liu
6994361550fb2c0be4783ea2
Hanieh Deilamsalehy
6994361550fb2c0be4783ea3
Nesreen K. Ahmed
6994361550fb2c0be4783ea4
Puneet Mathur
6994361550fb2c0be4783ea5
Nedim Lipka
6994361550fb2c0be4783ea6
Franck Dernoncourt
claimed_verified
2026-02-17T15:08:40.658Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
Franck-Dernoncourt
user
Franck-Dernoncourt
6994361550fb2c0be4783ea7
Ryan A. Rossi
Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization because they better capture semantic information, are better at reasoning, and are more robust to paraphrasing. However, LLM judges show biases for length and order among others, and are vulnerable to various adversarial input prompts. While recent studies have looked into these biases, few have analyzed them at a more granular level in relation to a well-defined overlap metric. In this work we provide an LLM judge bias analysis as a function of overlap with human-written responses in the domain of summarization. We test 9 recent LLMs with parameter counts ranging from 1 billion to 12 billion, including variants of Gemma 3 and LLaMA 3. We find that LLM judges increasingly prefer summaries generated by other LLMs over those written by humans as the similarities (as measured by ROUGE and BLEU) between the judged summaries decrease, and this pattern extends to all but one model tested, and exists regardless of the models' own position biases. Additionally, we find that models struggle to judge even summaries with limited overlaps, suggesting that LLM-as-a-judge in the summary domain should rely on techniques beyond a simple comparison.
LLM judges exhibit bias toward summaries similar to their own generation, with performance deteriorating as summary overlap with human references decreases across multiple model sizes and architectures.
large language model LLM judge summarization ROUGE BLEU overlap metric parameter count Gemma 3 LLaMA 3