I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution

TL;DR

The paper investigates leveraging large language models for source code authorship attribution to address generalization and labeled-data bottlenecks in software forensics. It systematically evaluates zero-shot and few-shot prompting across multiple LLM families, introducing a tournament prompting method to scale to hundreds of authors, achieving meaningful accuracy on large datasets. The study demonstrates cross-language generalization to Java, robustness to some adversarial attacks, and highlights the critical role of prompt design and model choice in practical deployment. These findings open pathways for applying LLMs to code forensics and software engineering tasks without task-specific fine-tuning, while underscoring considerations of cost, robustness, and prompt engineering.

Abstract

Source code authorship attribution is important in software forensics, plagiarism detection, and protecting software patch integrity. Existing techniques often rely on supervised machine learning, which struggles with generalization across different programming languages and coding styles due to the need for large labeled datasets. Inspired by recent advances in natural language authorship analysis using large language models (LLMs), which have shown exceptional performance without task-specific tuning, this paper explores the use of LLMs for source code authorship attribution. We present a comprehensive study demonstrating that state-of-the-art LLMs can successfully attribute source code authorship across different languages. LLMs can determine whether two code snippets are written by the same author with zero-shot prompting, achieving a Matthews Correlation Coefficient (MCC) of 0.78, and can attribute code authorship from a small set of reference code snippets via few-shot learning, achieving MCC of 0.77. Additionally, LLMs show some adversarial robustness against misattribution attacks. Despite these capabilities, we found that naive prompting of LLMs does not scale well with a large number of authors due to input token limitations. To address this, we propose a tournament-style approach for large-scale attribution. Evaluating this approach on datasets of C++ (500 authors, 26,355 samples) and Java (686 authors, 55,267 samples) code from GitHub, we achieve classification accuracy of up to 65% for C++ and 68.7% for Java using only one reference per author. These results open new possibilities for applying LLMs to code authorship attribution in cybersecurity and software engineering.

Figures (5)

• Figure 1: An overview of LLM-based code authorship attribution.
• Figure 2: Prompt templates (Simple, Detailed, and Complex) for the zero-shot code authorship verification experiment.
• Figure 3: Prompt templates (Simple, Detailed, and Complex) for the few-shot (one-, two-, and three-shot) code authorship attribution experiment.
• Figure 4: Overview of tournament prompting.
• Figure 5: Prompt templates Simple for the code authorship experiment with tournament prompting.