Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation.
698170a6ce18b18628096010
Zhuochun Li
claimed_verified
2026-02-04T12:33:14.224Z
65550ee2f4f9996af318cd36
/avatars/1da67f8e1ab5ecde66e99aae00b42a0b.svg
Zhuochun Li
zhuochun
user
zhuochun
698170a6ce18b18628096011
Yong Zhang
claimed_verified
2026-02-03T10:07:38.423Z
65543181ace6c512cf829cf9
/avatars/8360cb3e43ed25dc55d77c900ece1a0a.svg
ZHANG YONG
ReRaWo
user
ReRaWo
698170a6ce18b18628096012
Ming Li
698170a6ce18b18628096013
Yuelyu Ji
698170a6ce18b18628096014
Yiming Zeng
698170a6ce18b18628096015
Ning Cheng
698170a6ce18b18628096016
Yun Zhu
698170a6ce18b18628096017
Yanmeng Wang
698170a6ce18b18628096018
Shaojun Wang
698170a6ce18b18628096019
Jing Xiao
698170a6ce18b1862809601a
Daqing He
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Large language models (LLMs) are widely used as reference-free evaluators via prompting, but this "LLM-as-a-Judge" paradigm is costly, opaque, and sensitive to prompt design. In this work, we investigate whether smaller models can serve as efficient evaluators by leveraging internal representations instead of surface generation. We uncover a consistent empirical pattern: small LMs, despite with weak generative ability, encode rich evaluative signals in their hidden states. This motivates us to propose the Semantic Capacity Asymmetry Hypothesis: evaluation requires significantly less semantic capacity than generation and can be grounded in intermediate representations, suggesting that evaluation does not necessarily need to rely on large-scale generative models but can instead leverage latent features from smaller ones. Our findings motivate a paradigm shift from LLM-as-a-Judge to Representation-as-a-Judge, a decoding-free evaluation strategy that probes internal model structure rather than relying on prompted output. We instantiate this paradigm through INSPECTOR, a probing-based framework that predicts aspect-level evaluation scores from small model representations. Experiments on reasoning benchmarks (GSM8K, MATH, GPQA) show that INSPECTOR substantially outperforms prompting-based small LMs and closely approximates full LLM judges, while offering a more efficient, reliable, and interpretable alternative for scalable evaluation.
Small language models can effectively evaluate outputs by leveraging internal representations rather than generating responses, enabling a more efficient and interpretable evaluation approach through a probing-based framework.
large language models small language models hidden states semantic capacity representation learning LLM-as-a-Judge Representation-as-a-Judge probing-based framework aspect-level evaluation internal representations