References Improve LLM Alignment in Non-Verifiable Domains
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
6997e5f37a658569d5a10217
Kejian Shi
admin_assigned
2026-02-20T09:09:51.993Z
61993bc305b9430e5369db41
https://cdn-avatars.huggingface.co/v1/production/uploads/61993bc305b9430e5369db41/YmLB-UlaeCHhp5QHxNga-.png
Kejian Shi
kejian
user
kejian
6997e5f37a658569d5a10218
Yixin Liu
admin_assigned
2026-02-20T09:10:11.122Z
6244de1c1c560fb11edfca44
/avatars/36558928bd04be7f49837d4c603681d7.svg
Yixin Liu
henryL7
user
henryL7
6997e5f37a658569d5a10219
Peifeng Wang
admin_assigned
2026-02-20T09:10:17.382Z
680fd00a1ae38d47c1670ef2
/avatars/31610878448dce5032ab9305f2b87643.svg
Peifeng Wang
peifengw
user
peifengw
6997e5f37a658569d5a1021a
Alexander R. Fabbri
6997e5f37a658569d5a1021b
Shafiq Joty
6997e5f37a658569d5a1021c
Arman Cohan
admin_assigned
2026-02-20T09:10:33.361Z
5f5ba21188f57f65f951f255
https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
Arman Cohan
armanc
user
armanc
References Improve LLM Alignment in Non-Verifiable Domains
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Reference-guided LLM-evaluators enhance LLM alignment by serving as soft verifiers, improving judge accuracy and enabling effective post-training in non-verifiable domains through self-improvement techniques.
Reinforcement Learning with Verifiable Rewards LLM alignment reference-guided evaluators LLM judges self-improvement SFT ArmoRM AlpacaEval Arena-Hard