Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fingerprinting AI Coding Agents on GitHub

Taher A. Ghaleb

TL;DR

This work tackles the problem of attributing code authorship to AI coding agents when PRs are submitted under human accounts. It develops a fingerprinting approach using 41 features spanning commit messages, PR structure, and code characteristics, evaluated on 33,580 PRs from five agents with a high 97.2% F1-score for multi-class identification via XGBoost. The study reveals distinct, agent-specific fingerprints—such as Codex's multiline commits and Claude Code's dense conditionals and comments—offering practical utility for repository governance and dataset integrity in empirical software engineering. These findings have significant implications for policy enforcement, agent design, and research validity, while also highlighting limitations like minority-class recall and potential adversarial evasion, guiding future work on robustness and broader agent coverage.

Abstract

AI coding agents are reshaping software development through both autonomous and human-mediated pull requests (PRs). When developers use AI agents to generate code under their own accounts, code authorship attribution becomes critical for repository governance, research validity, and understanding modern development practices. We present the first study on fingerprinting AI coding agents, analyzing 33,580 PRs from five major agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, Claude Code) to identify behavioral signatures. With 41 features spanning commit messages, PR structure, and code characteristics, we achieve 97.2% F1-score in multi-class agent identification. We uncover distinct fingerprints: Codex shows unique multiline commit patterns (67.5% feature importance), and Claude Code exhibits distinctive code structure (27.2% importance of conditional statements). These signatures reveal that AI coding tools produce detectable behavioral patterns, suggesting potential for identifying AI contributions in software repositories.

Fingerprinting AI Coding Agents on GitHub

TL;DR

This work tackles the problem of attributing code authorship to AI coding agents when PRs are submitted under human accounts. It develops a fingerprinting approach using 41 features spanning commit messages, PR structure, and code characteristics, evaluated on 33,580 PRs from five agents with a high 97.2% F1-score for multi-class identification via XGBoost. The study reveals distinct, agent-specific fingerprints—such as Codex's multiline commits and Claude Code's dense conditionals and comments—offering practical utility for repository governance and dataset integrity in empirical software engineering. These findings have significant implications for policy enforcement, agent design, and research validity, while also highlighting limitations like minority-class recall and potential adversarial evasion, guiding future work on robustness and broader agent coverage.

Abstract

AI coding agents are reshaping software development through both autonomous and human-mediated pull requests (PRs). When developers use AI agents to generate code under their own accounts, code authorship attribution becomes critical for repository governance, research validity, and understanding modern development practices. We present the first study on fingerprinting AI coding agents, analyzing 33,580 PRs from five major agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, Claude Code) to identify behavioral signatures. With 41 features spanning commit messages, PR structure, and code characteristics, we achieve 97.2% F1-score in multi-class agent identification. We uncover distinct fingerprints: Codex shows unique multiline commit patterns (67.5% feature importance), and Claude Code exhibits distinctive code structure (27.2% importance of conditional statements). These signatures reveal that AI coding tools produce detectable behavioral patterns, suggesting potential for identifying AI contributions in software repositories.
Paper Structure (22 sections, 2 figures, 2 tables)

This paper contains 22 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Empirical Analysis and Results
  3. Data Source
  4. Feature Engineering
  5. Feature Extraction
  6. Feature Reduction
  7. Classification Methodology
  8. Multi-class classification
  9. One-vs-rest binary classification
  10. Hyperparameters
  11. RQ1: How accurately can we identify which AI coding agent submitted a pull request?
  12. Motivation.
  13. Approach.
  14. Findings.
  15. RQ2: What characteristics distinguish AI coding agents from each other?
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Confusion matrix for XGBoost multi-class agent classification. Red boxes mark correct predictions.
  • Figure 2: Agent-specific feature importance from one-vs-rest XGBoost models.