Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting RLVR Training Data via Structural Convergence of Reasoning

Hongbo Zhang, Yue Yang, Jianhao Yan, Guangsheng Bao, Yue Zhang, Yue Zhang

TL;DR

This paper shows that reinforcement learning with verifiable rewards (RLVR) reshapes model reasoning by collapsing diverse trajectories into a small set of structural modes, especially in symbolic components, creating a distinct exposure signal for RLVR data. It introduces Min-$k$NN Distance, a fully black-box detector that detects RLVR exposure by measuring the structural similarity of multiple completions for a given prompt, without relying on token probabilities or reference models. Across multiple models, RLVR algorithms, and data sources, Min-$k$NN Distance consistently outperforms baselines, demonstrating robustness to paraphrase, distillation, and cross-domain data. The work contributes a practical, model-agnostic tool for RLVR-data detection, with implications for transparency and reliability in benchmark evaluations and post-training data usage.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-$k$NN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the $k$ smallest nearest-neighbor edit distances. Min-$k$NN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-$k$NN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.

Detecting RLVR Training Data via Structural Convergence of Reasoning

TL;DR

This paper shows that reinforcement learning with verifiable rewards (RLVR) reshapes model reasoning by collapsing diverse trajectories into a small set of structural modes, especially in symbolic components, creating a distinct exposure signal for RLVR data. It introduces Min-NN Distance, a fully black-box detector that detects RLVR exposure by measuring the structural similarity of multiple completions for a given prompt, without relying on token probabilities or reference models. Across multiple models, RLVR algorithms, and data sources, Min-NN Distance consistently outperforms baselines, demonstrating robustness to paraphrase, distillation, and cross-domain data. The work contributes a practical, model-agnostic tool for RLVR-data detection, with implications for transparency and reliability in benchmark evaluations and post-training data usage.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-NN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the smallest nearest-neighbor edit distances. Min-NN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-NN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.
Paper Structure (37 sections, 7 equations, 9 figures, 9 tables)

This paper contains 37 sections, 7 equations, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Problem Definition:
  4. RLVR Training Process:
  5. Analyzing Reasoning Pattern under RLVR
  6. RLVR Induces Generation Rigidity.
  7. RLVR Converges Symbolic Reasoning Segments.
  8. RLVR Converges Reasoning into a Limited Set of Structural Modes.
  9. Greater Rigidity in Seen Data Compared to Unseen Data.
  10. Min-$k$NN Distance : A Simple Black-Box Detector for RLVR Exposure
  11. Experiments
  12. Experimental Setup
  13. Models and Datasets
  14. Baselines
  15. Implementation and Evaluation Metrics
  16. ...and 22 more sections

Figures (9)

  • Figure 1: We study how RLVR induces structural convergence in reasoning trajectories. Left: RLVR compresses diverse reasoning paths into shared structural modes. Middle: multiple completions cluster into few reasoning structures. Right: Min-$k$NN Distance quantifies this collapse via nearest-neighbor edit distances, yielding low values for seen prompts and high values for unseen prompts.
  • Figure 2: Evolution of generation diversity during RLVR training under DAPO and GRPO, measured in a per-input setting by three complementary metrics capturing lexical (EAD), logical (NLI), and semantic (embedding) diversity.
  • Figure 3: Rigid 3-gram category counts in GRPO training checkpoints. GPT-4o labeled rigid 3-grams into four categories.
  • Figure 4: Heatmap of rigid 3-grams that appear in at least three completions for a single prompt. The clustering is achieved via hierarchical agglomerative clustering.
  • Figure 5: Distribution of the number of reasoning structure clusters across 50 training prompts.
  • ...and 4 more figures