ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods either treat all tests equally or rely on ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a circular dependency. Our key insight is that we need not determine test correctness at all: test votes should rank, not merely count. What matters is not how many codes pass a test, but whether the test can distinguish correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC~(LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose ACES~(AUC ConsistEncy Scoring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@k on multiple code generation benchmarks.
69d463e32c523d36431fb022
Hui Sun
claimed_verified
2026-04-07T08:34:49.079Z
69d26ea1fb7d4cfbfac10a64
/avatars/4b6c95c785db83cec70cd5d2117a4104.svg
Hui Sun
sun0o0
user
sun0o0
69d463e32c523d36431fb023
Yun-Ji Zhang
69d463e32c523d36431fb024
Zheng Xie
69d463e32c523d36431fb025
Ren-Biao Liu
claimed_verified
2026-04-08T16:34:25.055Z
6522100706fa58638b6af76f
https://cdn-avatars.huggingface.co/v1/production/uploads/6522100706fa58638b6af76f/uCTgHOzxKZIQAIf7oD0kv.png
Ren-Biao Liu
richardodliu
user
richardodliu
69d463e32c523d36431fb026
Yali Du
69d463e32c523d36431fb027
Xin-Ye Li
69d463e32c523d36431fb028
Ming Li
ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Selecting LLM-generated code candidates using LLM-generated tests is challenging because the tests themselves may be incorrect. Existing methods either treat all tests equally or rely on ad-hoc heuristics to filter unreliable tests. Yet determining test correctness requires knowing which codes are correct, creating a circular dependency. Our key insight is that we need not determine test correctness at all: test votes should rank, not merely count. What matters is not how many codes pass a test, but whether the test can distinguish correct from incorrect code. We break the circular dependency via leave-one-out evaluation: hold out one test, rank codes by their aggregate scores on all remaining tests, and measure whether the held-out test's pass/fail pattern agrees with this ranking. We formalize this agreement as the leave-one-out AUC~(LOO-AUC) and prove that the expected LOO-AUC is proportional to each test's ability to separate correct code from incorrect code. Building on this, we propose ACES~(AUC ConsistEncy Scoring) with two complementary variants: ACES-C provides closed-form weights that provably approximate the oracle in expectation under a mild assumption on average test quality; ACES-O drops this assumption and iteratively optimizes a differentiable LOO-AUC objective. Both operate solely on the binary pass matrix with negligible overhead, and achieve state-of-the-art Pass@k on multiple code generation benchmarks.
Researchers address the challenge of selecting correct code candidates from LLM-generated outputs by developing ACES, a method that ranks tests based on their ability to distinguish correct from incorrect code through leave-one-out evaluation and AUC consistency scoring.
LLM-generated code test correctness circular dependency leave-one-out evaluation LOO-AUC AUC consistency scoring Pass@k binary pass matrix oracle approximation