Are Large Reasoning Models Good Translation Evaluators? Analysis and
  Performance Boost
Recent advancements in large reasoning models (LRMs) have introduced an
intermediate "thinking" process prior to generating final answers, improving
their reasoning capabilities on complex downstream tasks. However, the
potential of LRMs as evaluators for machine translation (MT) quality remains
underexplored. We provides the first systematic analysis of LRM-as-a-judge in
MT evaluation. We identify key challenges, revealing LRMs require tailored
evaluation materials, tend to "overthink" simpler instances and have issues
with scoring mechanisms leading to overestimation. To address these, we propose
to calibrate LRM thinking by training them on synthetic, human-like thinking
trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this
approach largely reduces thinking budgets by ~35x while concurrently improving
evaluation performance across different LRM scales from 7B to 32B (e.g.,
R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These
findings highlight the potential of efficiently calibrated LRMs to advance
fine-grained automatic MT evaluation.
68fb8ba56cdff8b857f46b90
Runzhe Zhan
claimed_verified
2025-10-27T10:29:02.663Z
62495cb96ee7ee6b646db130
https://cdn-avatars.huggingface.co/v1/production/uploads/62495cb96ee7ee6b646db130/UwBXmvcMq7LMvBWUw0xo3.jpeg
Runzhe Zhan
rzzhan
user
rzzhan
68fb8ba56cdff8b857f46b91
Zhihong Huang
68fb8ba56cdff8b857f46b92
Xinyi Yang
68fb8ba56cdff8b857f46b93
Lidia S. Chao
68fb8ba56cdff8b857f46b94
Min Yang
68fb8ba56cdff8b857f46b95
Derek F. Wong
Are Large Reasoning Models Good Translation Evaluators? Analysis and
  Performance Boost
Recent advancements in large reasoning models (LRMs) have introduced an
intermediate "thinking" process prior to generating final answers, improving
their reasoning capabilities on complex downstream tasks. However, the
potential of LRMs as evaluators for machine translation (MT) quality remains
underexplored. We provides the first systematic analysis of LRM-as-a-judge in
MT evaluation. We identify key challenges, revealing LRMs require tailored
evaluation materials, tend to "overthink" simpler instances and have issues
with scoring mechanisms leading to overestimation. To address these, we propose
to calibrate LRM thinking by training them on synthetic, human-like thinking
trajectories. Our experiments on WMT24 Metrics benchmarks demonstrate that this
approach largely reduces thinking budgets by ~35x while concurrently improving
evaluation performance across different LRM scales from 7B to 32B (e.g.,
R1-Distill-Qwen-7B achieves a +8.7 correlation point improvement). These
findings highlight the potential of efficiently calibrated LRMs to advance
fine-grained automatic MT evaluation.
Calibrating large reasoning models with synthetic human-like thinking trajectories improves their efficiency and performance in machine translation evaluation.
large reasoning models thinking process machine translation evaluation synthetic thinking trajectories WMT24 Metrics correlation point improvement