Detecting RLVR Training Data via Structural Convergence of Reasoning
Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-kNN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the k smallest nearest-neighbor edit distances. Min-kNN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-kNN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.
698ecc34cace060ff123aef5
Hongbo Zhang
698ecc34cace060ff123aef6
Yue Yang
698ecc34cace060ff123aef7
Jianhao Yan
698ecc34cace060ff123aef8
Guangsheng Bao
698ecc34cace060ff123aef9
Yue Zhang
698ecc34cace060ff123aefa
Yue Zhang
Detecting RLVR Training Data via Structural Convergence of Reasoning
Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-kNN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the k smallest nearest-neighbor edit distances. Min-kNN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-kNN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.
Reinforcement learning with verifiable rewards induces behavioral signatures that can be detected using a black-box method based on prompt generation diversity, outperforming existing contamination detection approaches.
reinforcement learning with verifiable rewards reward feedback self-generated reasoning trajectories membership inference RL contamination detection Min-kNN Distance behavioral signature