RM -RF: Reward Model for Run-Free Unit Test Evaluation
We present RM-RF, a lightweight reward model for run-free evaluation of automatically generated unit tests. Instead of repeatedly compiling and executing candidate tests, RM-RF predicts - from source and test code alone - three execution-derived signals: (1) whether the augmented test suite compiles and runs successfully, (2) whether the generated test cases increase code coverage, and (3) whether the generated test cases improve the mutation kill rate. To train and evaluate RM-RF we assemble a multilingual dataset (Java, Python, Go) of focal files, test files, and candidate test additions labeled by an execution-based pipeline, and we release an associated dataset and methodology for comparative evaluation. We tested multiple model families and tuning regimes (zero-shot, full fine-tuning, and PEFT via LoRA), achieving an average F1 of 0.69 across the three targets. Compared to conventional compile-and-run instruments, RM-RF provides substantially lower latency and infrastructure cost while delivering competitive predictive fidelity, enabling fast, scalable feedback for large-scale test generation and RL-based code optimization.
6976fdd85d41524304c1367e
Elena Bruches
claimed_verified
2026-02-02T17:01:03.322Z
66728993cc71f0dce43fb93a
https://cdn-avatars.huggingface.co/v1/production/uploads/66728993cc71f0dce43fb93a/3Lf8ltxKSDOhbQxOCt-cE.jpeg
Bruches Elena
brucheselena
user
brucheselena
6976fdd85d41524304c1367f
Daniil Grebenkin
claimed_verified
2026-01-26T08:28:32.856Z
63cb976d80ba2ca4151b67a2
https://cdn-avatars.huggingface.co/v1/production/uploads/1675713278440-63cb976d80ba2ca4151b67a2.jpeg
Daniel Grebenkin
dangrebenkin
user
dangrebenkin
6976fdd85d41524304c13680
Mikhail Klementev
6976fdd85d41524304c13681
Vadim Alperovich
6976fdd85d41524304c13682
Roman Derunets
claimed_verified
2026-01-26T08:28:34.729Z
6415cb01486c7c9a5d1560f3
https://cdn-avatars.huggingface.co/v1/production/uploads/6415cb01486c7c9a5d1560f3/tKQPhr1o-1SLSvKg3un6J.jpeg
Roman Derunets
rmndrnts
user
rmndrnts
6976fdd85d41524304c13683
Dari Baturova
claimed_verified
2026-02-02T17:01:01.399Z
63ea1ca8afddf881179de4fe
/avatars/2690766ee09eefff296bcdac46e87b3b.svg
Dari
doooori
user
doooori
6976fdd85d41524304c13684
Georgy Mkrtchyan
6976fdd85d41524304c13685
Oleg Sedukhin
claimed_verified
2026-02-23T09:47:40.810Z
61dd9daedb45389905634d3f
/avatars/a0482317b6118d37da294b9dc2bf5a39.svg
Oleg Sedukhin
greyzyablik
user
greyzyablik
6976fdd85d41524304c13686
Ivan Bondarenko
claimed_verified
2026-02-09T14:32:11.635Z
62b1e0f76a5435fd9a60a8dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1655824626110-noauth.png
Ivan Bondarenko
bond005
user
bond005
6976fdd85d41524304c13687
Nikolay Bushkov
6976fdd85d41524304c13688
Stanislav Moiseev
RM -RF: Reward Model for Run-Free Unit Test Evaluation
We present RM-RF, a lightweight reward model for run-free evaluation of automatically generated unit tests. Instead of repeatedly compiling and executing candidate tests, RM-RF predicts - from source and test code alone - three execution-derived signals: (1) whether the augmented test suite compiles and runs successfully, (2) whether the generated test cases increase code coverage, and (3) whether the generated test cases improve the mutation kill rate. To train and evaluate RM-RF we assemble a multilingual dataset (Java, Python, Go) of focal files, test files, and candidate test additions labeled by an execution-based pipeline, and we release an associated dataset and methodology for comparative evaluation. We tested multiple model families and tuning regimes (zero-shot, full fine-tuning, and PEFT via LoRA), achieving an average F1 of 0.69 across the three targets. Compared to conventional compile-and-run instruments, RM-RF provides substantially lower latency and infrastructure cost while delivering competitive predictive fidelity, enabling fast, scalable feedback for large-scale test generation and RL-based code optimization.
RM-RF is a lightweight reward model that predicts execution outcomes from source code alone, offering faster and more cost-effective evaluation than traditional compile-and-run methods.
reward model run-free evaluation unit tests execution-derived signals code coverage mutation kill rate multilingual dataset focal files test files candidate test additions execution-based pipeline model families tuning regimes zero-shot full fine-tuning PEFT LoRA F1 score