Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models

Farnaz Soltaniani, Mohammad Ghafari

TL;DR

The paper investigates how data leakage from duplication in SecretBench inflates secret-detection performance and distorts real-world generalization. It formalizes secret context, duplicates, and deduplication into three evaluation scenarios (Mixed, Near Duplicate, Unique) and tests four models, including GraphCodeBERT variants, LSTM, and Random Forest. Findings show substantial degradation in MCC when duplicates are removed, with GraphCodeBERT most robust and RF most vulnerable, highlighting memorization rather than true learning in prior reports. The work emphasizes dataset hygiene and robust evaluation protocols, and provides replication resources to facilitate validation and broader adoption of leakage-resistant methods.

Abstract

Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness.

From Data Leak to Secret Misses: The Impact of Data Leakage on Secret Detection Models

TL;DR

The paper investigates how data leakage from duplication in SecretBench inflates secret-detection performance and distorts real-world generalization. It formalizes secret context, duplicates, and deduplication into three evaluation scenarios (Mixed, Near Duplicate, Unique) and tests four models, including GraphCodeBERT variants, LSTM, and Random Forest. Findings show substantial degradation in MCC when duplicates are removed, with GraphCodeBERT most robust and RF most vulnerable, highlighting memorization rather than true learning in prior reports. The work emphasizes dataset hygiene and robust evaluation protocols, and provides replication resources to facilitate validation and broader adoption of leakage-resistant methods.

Abstract

Machine learning models are increasingly used for software security tasks. These models are commonly trained and evaluated on large Internet-derived datasets, which often contain duplicated or highly similar samples. When such samples are split across training and test sets, data leakage may occur, allowing models to memorize patterns instead of learning to generalize. We investigate duplication in a widely used benchmark dataset of hard coded secrets and show how data leakage can substantially inflate the reported performance of AI-based secret detectors, resulting in a misleading picture of their real-world effectiveness.
Paper Structure (23 sections, 4 equations, 3 figures, 6 tables)

This paper contains 23 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. INTRODUCTION
  2. Problem Statement
  3. Secret Context
  4. Deduplication Analysis
  5. Exact Duplicates ($C_{exact}$)
  6. Near Duplicates ($C_{near}$)
  7. Uniques ($C_{unique}$)
  8. Duplication in SecretBench
  9. Methodology
  10. Dataset
  11. Training and Testing
  12. Mixed (Baseline on Duplicated Data)
  13. Near Duplicate (Baseline on Deduplicated Data)
  14. Unique (Training on Duplicates, Testing on Unique Data)
  15. Models
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Example of a secret code context.
  • Figure 2: Three Main Train/Test Scenarios.
  • Figure 3: MCC performance across test scenarios.