Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tricky$^2$: Towards a Benchmark for Evaluating Human and LLM Error Interactions

Cole Granger, Dipin Khati, Daniel Rodriguez-Cardenas, Denys Poshyvanyk

TL;DR

Tricky2 addresses a gap in software engineering benchmarks by evaluating how human-written bugs and LLM-generated errors interact within the same codebase. The authors introduce a taxonomy-guided bug-injection workflow that extends TrickyBugs with LLM-originated faults across C++, Python, and Java, producing Human-only, LLM-only, and Human+LLM splits with identical test suites. They provide a small but concrete baseline across origin classification, error localization, and program repair, revealing that mixed-origin defects can degrade repair success and highlighting interaction effects not observable in single-origin benchmarks. The dataset and prompts are released publicly to support reproducibility and future exploration of robust, error-aware human–AI code collaboration.

Abstract

Large language models (LLMs) are increasingly integrated into software development workflows, yet they often introduce subtle logic or data-misuse errors that differ from human bugs. To study how these two error types interact, we construct Tricky$^2$, a hybrid dataset that augments the existing TrickyBugs corpus of human-written defects with errors injected by both GPT-5 and OpenAI-oss-20b across C++, Python, and Java programs. Our approach uses a taxonomy-guided prompting framework to generate machine-originated bugs while preserving original human defects and program structure. The resulting corpus spans human-only, LLM-only, and human+LLM splits, enabling analysis of mixed-origin error behavior, multi-bug repair robustness, and reliability in hybrid human-machine code. This paper outlines the dataset construction pipeline and illustrates its use through small-scale baseline evaluations of classification, localization, and repair tasks.

Tricky$^2$: Towards a Benchmark for Evaluating Human and LLM Error Interactions

TL;DR

Tricky2 addresses a gap in software engineering benchmarks by evaluating how human-written bugs and LLM-generated errors interact within the same codebase. The authors introduce a taxonomy-guided bug-injection workflow that extends TrickyBugs with LLM-originated faults across C++, Python, and Java, producing Human-only, LLM-only, and Human+LLM splits with identical test suites. They provide a small but concrete baseline across origin classification, error localization, and program repair, revealing that mixed-origin defects can degrade repair success and highlighting interaction effects not observable in single-origin benchmarks. The dataset and prompts are released publicly to support reproducibility and future exploration of robust, error-aware human–AI code collaboration.

Abstract

Large language models (LLMs) are increasingly integrated into software development workflows, yet they often introduce subtle logic or data-misuse errors that differ from human bugs. To study how these two error types interact, we construct Tricky, a hybrid dataset that augments the existing TrickyBugs corpus of human-written defects with errors injected by both GPT-5 and OpenAI-oss-20b across C++, Python, and Java programs. Our approach uses a taxonomy-guided prompting framework to generate machine-originated bugs while preserving original human defects and program structure. The resulting corpus spans human-only, LLM-only, and human+LLM splits, enabling analysis of mixed-origin error behavior, multi-bug repair robustness, and reliability in hybrid human-machine code. This paper outlines the dataset construction pipeline and illustrates its use through small-scale baseline evaluations of classification, localization, and repair tasks.
Paper Structure (14 sections, 3 figures)

This paper contains 14 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Prompt Design for Bug Injection
  4. Automated Generation and Validation Pipeline
  5. Metadata and Reproducibility
  6. Baseline Evaluation Across Splits and Tasks
  7. Task 1: Origin Classification
  8. Task 2: Error Identification
  9. Task 3: Program Repair
  10. Reporting
  11. Discussion
  12. Limitations
  13. Conclusions & Future Work
  14. Data Availability

Figures (3)

  • Figure 1: Construction of the Tricky2 benchmark from the TrickyBugs dataset. Buggy human submissions form the Human-only split, error-injected buggy programs yield the Human+LLM split, and error-injected fixed programs yield the LLM-only split. Each injection applies a controlled taxonomy. All splits share identical test cases and fixed references.
  • Figure 2: Distribution of the injected errors by taxonomy level and split (Human+LLM vs. LLM-Only). There are fewer problems in the LLM-Only versions because there are not fixed programs for each buggy program in the original dataset. The taxonomy was provided with each injection prompt, but total numbers of each category were not provided to the model.
  • Figure 3: Baseline evaluation visualizations: (Left) predicted origin counts; (Middle) repair outcomes; (Right) taxonomy confusion heatmap.