Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RepoMark: A Data-Usage Auditing Framework for Code Large Language Models

Wenjie Qu, Yuguang Zhou, Bo Wang, Yuexin Li, Lionel Z. Wang, Jinyuan Jia, Jiaheng Zhang

TL;DR

RepoMark tackles the ethical and legal challenges of training code LLMs on public repositories by introducing a data-usage auditing framework based on semantic-preserving, imperceptible code marking. It generates $m$ semantically equivalent variants per file, publishes one at random, and uses a rank-based statistic $S$ to test whether a repository was used in training, with a provable FDR bound. Empirically, RepoMark achieves a detection success rate exceeding $90\%$ at a $5\%$ FDR on small repositories across multiple code models and datasets, while preserving code quality and showing robustness to countermeasures. By enabling authors to verify training usage, RepoMark advances transparency and accountability in code LLM ecosystems, and its variable-renaming marking strategy contributes practical robustness for real-world audits.

Abstract

The rapid development of Large Language Models (LLMs) for code generation has transformed software development by automating coding tasks with unprecedented efficiency. However, the training of these models on open-source code repositories (e.g., from GitHub) raises critical ethical and legal concerns, particularly regarding data authorization and open-source license compliance. Developers are increasingly questioning whether model trainers have obtained proper authorization before using repositories for training, especially given the lack of transparency in data collection. To address these concerns, we propose a novel data marking framework RepoMark to audit the data usage of code LLMs. Our method enables auditors to verify whether their code has been used in training, while ensuring semantic preservation, imperceptibility, and theoretical false detection rate (FDR) guarantees. By generating multiple semantically equivalent code variants, RepoMark introduces data marks into the code files, and during detection, RepoMark leverages a novel ranking-based hypothesis test to detect model behavior difference on trained data. Compared to prior data auditing approaches, RepoMark significantly enhances data efficiency, allowing effective auditing even when the user's repository possesses only a small number of code files. Experiments demonstrate that RepoMark achieves a detection success rate over 90\% on small code repositories under a strict FDR guarantee of 5\%. This represents a significant advancement over existing data marking techniques, all of which only achieve accuracy below 55\% under identical settings. This further validates RepoMark as a robust, theoretically sound, and promising solution for enhancing transparency in code LLM training, which can safeguard the rights of code authors.

RepoMark: A Data-Usage Auditing Framework for Code Large Language Models

TL;DR

RepoMark tackles the ethical and legal challenges of training code LLMs on public repositories by introducing a data-usage auditing framework based on semantic-preserving, imperceptible code marking. It generates semantically equivalent variants per file, publishes one at random, and uses a rank-based statistic to test whether a repository was used in training, with a provable FDR bound. Empirically, RepoMark achieves a detection success rate exceeding at a FDR on small repositories across multiple code models and datasets, while preserving code quality and showing robustness to countermeasures. By enabling authors to verify training usage, RepoMark advances transparency and accountability in code LLM ecosystems, and its variable-renaming marking strategy contributes practical robustness for real-world audits.

Abstract

The rapid development of Large Language Models (LLMs) for code generation has transformed software development by automating coding tasks with unprecedented efficiency. However, the training of these models on open-source code repositories (e.g., from GitHub) raises critical ethical and legal concerns, particularly regarding data authorization and open-source license compliance. Developers are increasingly questioning whether model trainers have obtained proper authorization before using repositories for training, especially given the lack of transparency in data collection. To address these concerns, we propose a novel data marking framework RepoMark to audit the data usage of code LLMs. Our method enables auditors to verify whether their code has been used in training, while ensuring semantic preservation, imperceptibility, and theoretical false detection rate (FDR) guarantees. By generating multiple semantically equivalent code variants, RepoMark introduces data marks into the code files, and during detection, RepoMark leverages a novel ranking-based hypothesis test to detect model behavior difference on trained data. Compared to prior data auditing approaches, RepoMark significantly enhances data efficiency, allowing effective auditing even when the user's repository possesses only a small number of code files. Experiments demonstrate that RepoMark achieves a detection success rate over 90\% on small code repositories under a strict FDR guarantee of 5\%. This represents a significant advancement over existing data marking techniques, all of which only achieve accuracy below 55\% under identical settings. This further validates RepoMark as a robust, theoretically sound, and promising solution for enhancing transparency in code LLM training, which can safeguard the rights of code authors.

Paper Structure

This paper contains 23 sections, 1 theorem, 7 equations, 6 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Code large language models
  4. Membership inference attack
  5. Data marking
  6. Backdoor marking
  7. Contrastive marking
  8. Problem formulation
  9. Threat model
  10. Design goals
  11. Methodology
  12. New auditing framework
  13. Semantic preservation and imperceptible code marking
  14. Detection with FDR guarantee
  15. Further improving the data efficiency of RepoMark
  16. ...and 8 more sections

Key Result

Theorem 1

Under $H_0$, $\mathsf{rk}(\{f(w,x,t^1),\cdots, f(w,x,t^m)\}$$, f(w,x,t^r))$ is uniformly distributed among $\{1,\cdots,m\}$.

Figures (6)

  • Figure 1: The examples of code marked with RepoMark.
  • Figure 2: Illustration of RepoMark's marking process for code. This simplified figure highlights a single file with one variable modification for clarity. In practice, RepoMark marks every file in the repository.
  • Figure 3: The distribution of the ratio of rank sum to $n*m$ under $H_0$ (the untrained case). With larger $n$, the distribution of the rank sum is more concentrated to $\frac{n(m+1)}{2}$, under $H_0$.
  • Figure 4: DSRs of RepoMark across 3 models and 3 datasets under FDR guarantee levels of 1%, 2%, 5%, 10%, 20%.
  • Figure 5: Impact of version number $m$, mark sparsity parameter $K$, repository file number $N$ on the DSR curve.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof