Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hidden DNA of LLM-Generated JavaScript: Structural Patterns Enable High-Accuracy Authorship Attribution

Norbert Tihanyi, Bilel Cherif, Richard A. Dubniczky, Mohamed Amine Ferrag, Tamás Bisztray

TL;DR

The paper tackles the problem of attributing JavaScript code generated by large language models to the specific producing model, arguing for persistent model fingerprints beyond AI-vs-human detection. It introduces the LLM-NodeJS dataset and a specialized encoder-based CodeT5-JSA classifier to enable large-scale, transformation-robust authorship attribution across up to 20 classes. Key findings show high accuracy (around 95% for five-class, ~90% for five-class within-family, ~94% for ten-class, and ~88% for twenty-class) and robustness to code variants such as minification and obfuscation, with dataflow and AST/JSIR representations capturing the strongest signals. The work advances AI provenance and forensics by demonstrating that deep structural and semantic cues harden attribution against common code transformations and model evolution, and it provides open resources to support reproducibility and broader research in this domain.

Abstract

In this paper, we present the first large-scale study exploring whether JavaScript code generated by Large Language Models (LLMs) can reveal which model produced it, enabling reliable authorship attribution and model fingerprinting. With the rapid rise of AI-generated code, attribution is playing a critical role in detecting vulnerabilities, flagging malicious content, and ensuring accountability. While AI-vs-human detection usually treats AI as a single category we show that individual LLMs leave unique stylistic signatures, even among models belonging to the same family or parameter size. To this end, we introduce LLM-NodeJS, a dataset of 50,000 Node.js back-end programs from 20 large language models. Each has four transformed variants, yielding 250,000 unique JavaScript samples and two additional representations (JSIR and AST) for diverse research applications. Using this dataset, we benchmark traditional machine learning classifiers against fine-tuned Transformer encoders and introduce CodeT5-JSA, a custom architecture derived from the 770M-parameter CodeT5 model with its decoder removed and a modified classification head. It achieves 95.8% accuracy on five-class attribution, 94.6% on ten-class, and 88.5% on twenty-class tasks, surpassing other tested models such as BERT, CodeBERT, and Longformer. We demonstrate that classifiers capture deeper stylistic regularities in program dataflow and structure, rather than relying on surface-level features. As a result, attribution remains effective even after mangling, comment removal, and heavy code transformations. To support open science and reproducibility, we release the LLM-NodeJS dataset, Google Colab training scripts, and all related materials on GitHub: https://github.com/LLM-NodeJS-dataset.

The Hidden DNA of LLM-Generated JavaScript: Structural Patterns Enable High-Accuracy Authorship Attribution

TL;DR

The paper tackles the problem of attributing JavaScript code generated by large language models to the specific producing model, arguing for persistent model fingerprints beyond AI-vs-human detection. It introduces the LLM-NodeJS dataset and a specialized encoder-based CodeT5-JSA classifier to enable large-scale, transformation-robust authorship attribution across up to 20 classes. Key findings show high accuracy (around 95% for five-class, ~90% for five-class within-family, ~94% for ten-class, and ~88% for twenty-class) and robustness to code variants such as minification and obfuscation, with dataflow and AST/JSIR representations capturing the strongest signals. The work advances AI provenance and forensics by demonstrating that deep structural and semantic cues harden attribution against common code transformations and model evolution, and it provides open resources to support reproducibility and broader research in this domain.

Abstract

In this paper, we present the first large-scale study exploring whether JavaScript code generated by Large Language Models (LLMs) can reveal which model produced it, enabling reliable authorship attribution and model fingerprinting. With the rapid rise of AI-generated code, attribution is playing a critical role in detecting vulnerabilities, flagging malicious content, and ensuring accountability. While AI-vs-human detection usually treats AI as a single category we show that individual LLMs leave unique stylistic signatures, even among models belonging to the same family or parameter size. To this end, we introduce LLM-NodeJS, a dataset of 50,000 Node.js back-end programs from 20 large language models. Each has four transformed variants, yielding 250,000 unique JavaScript samples and two additional representations (JSIR and AST) for diverse research applications. Using this dataset, we benchmark traditional machine learning classifiers against fine-tuned Transformer encoders and introduce CodeT5-JSA, a custom architecture derived from the 770M-parameter CodeT5 model with its decoder removed and a modified classification head. It achieves 95.8% accuracy on five-class attribution, 94.6% on ten-class, and 88.5% on twenty-class tasks, surpassing other tested models such as BERT, CodeBERT, and Longformer. We demonstrate that classifiers capture deeper stylistic regularities in program dataflow and structure, rather than relying on surface-level features. As a result, attribution remains effective even after mangling, comment removal, and heavy code transformations. To support open science and reproducibility, we release the LLM-NodeJS dataset, Google Colab training scripts, and all related materials on GitHub: https://github.com/LLM-NodeJS-dataset.

Paper Structure

This paper contains 31 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Authorship attribution architectures
  4. Traditional Machine Learning
  5. Deep Learning
  6. Pre-trained and Generative Models
  7. Code Representation
  8. Feature Engineering Approaches
  9. Graph and Structural Representations
  10. Obfuscated Code Attribution
  11. Execution and Binary-Level Approaches
  12. Datasets, Benchmarks and Tasks
  13. Methodology
  14. Dataset Creation- The LLM-NodeJS dataset
  15. Selected Models for Dataset Evaluation
  16. ...and 16 more sections

Figures (3)

  • Figure 1: The seven-step methodology for LLM authorship attribution in JavaScript code.
  • Figure 2: Tokenized Dataset Statistics, where PX marks that X percentage of samples fall below that token count.
  • Figure 3: Confusion matrix for five-class attribution within the GPT family using BERT$_B$ on js_original