Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SnipGen: A Mining Repository Framework for Evaluating LLMs for Code

Daniel Rodriguez-Cardenas, Alejandro Velasco, Denys Poshyvanyk

TL;DR

SnipGen tackles the data-contamination problem in evaluating LLMs for code by mining recent GitHub commits to assemble fresh, method-level testbeds augmented with tailored prompts. It integrates a seven-dimensional data-point schema with eight prompt templates, enabling single- and multi-step prompting across code completion, commit generation, and code summarization. The framework generates six SE-task testbeds from ~227K data points and demonstrates practical utility through case studies (Galeras, SyntaxEval, ASTxplainer), highlighting how prompt design and model characteristics influence performance and explainability. By releasing the framework, generated testbeds, and prompt-generated datasets, SnipGen provides a reproducible, contamination-aware evaluation pipeline for advancing robust LLM assessment in software engineering contexts.

Abstract

Language Models (LLMs), such as transformer-based neural networks trained on billions of parameters, have become increasingly prevalent in software engineering (SE). These models, trained on extensive datasets that include code repositories, exhibit remarkable capabilities for SE tasks. However, evaluating their effectiveness poses significant challenges, primarily due to the potential overlap between the datasets used for training and those employed for evaluation. To address this issue, we introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation. SnipGen aims to mitigate data contamination by generating robust testbeds and crafting tailored data points to assist researchers and practitioners in evaluating LLMs for code-related tasks. In our exploratory study, SnipGen mined approximately 227K data points from 338K recent code changes in GitHub commits, focusing on method-level granularity. SnipGen features a collection of prompt templates that can be combined to create a Chain-of-Thought-like sequence of prompts, enabling a nuanced assessment of LLMs' code generation quality. By providing the mining tool, the methodology, and the dataset, SnipGen empowers researchers and practitioners to rigorously evaluate and interpret LLMs' performance in software engineering contexts.

SnipGen: A Mining Repository Framework for Evaluating LLMs for Code

TL;DR

SnipGen tackles the data-contamination problem in evaluating LLMs for code by mining recent GitHub commits to assemble fresh, method-level testbeds augmented with tailored prompts. It integrates a seven-dimensional data-point schema with eight prompt templates, enabling single- and multi-step prompting across code completion, commit generation, and code summarization. The framework generates six SE-task testbeds from ~227K data points and demonstrates practical utility through case studies (Galeras, SyntaxEval, ASTxplainer), highlighting how prompt design and model characteristics influence performance and explainability. By releasing the framework, generated testbeds, and prompt-generated datasets, SnipGen provides a reproducible, contamination-aware evaluation pipeline for advancing robust LLM assessment in software engineering contexts.

Abstract

Language Models (LLMs), such as transformer-based neural networks trained on billions of parameters, have become increasingly prevalent in software engineering (SE). These models, trained on extensive datasets that include code repositories, exhibit remarkable capabilities for SE tasks. However, evaluating their effectiveness poses significant challenges, primarily due to the potential overlap between the datasets used for training and those employed for evaluation. To address this issue, we introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation. SnipGen aims to mitigate data contamination by generating robust testbeds and crafting tailored data points to assist researchers and practitioners in evaluating LLMs for code-related tasks. In our exploratory study, SnipGen mined approximately 227K data points from 338K recent code changes in GitHub commits, focusing on method-level granularity. SnipGen features a collection of prompt templates that can be combined to create a Chain-of-Thought-like sequence of prompts, enabling a nuanced assessment of LLMs' code generation quality. By providing the mining tool, the methodology, and the dataset, SnipGen empowers researchers and practitioners to rigorously evaluate and interpret LLMs' performance in software engineering contexts.

Paper Structure

This paper contains 11 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The SnipGen Framework
  3. Data Point Feature Structure
  4. Software Engineering Tesbed Task
  5. Prompt templates
  6. SnipGen Prompt Generation and Use
  7. Experience Report
  8. SnipGen Testbed Generation
  9. Successful Use Cases
  10. Similar Datasets
  11. Limitations and future work

Figures (3)

  • Figure 1: SnipGen Data collection and prompt generation
  • Figure 2: SnipGen data schema. The snippet represents the core commit collected with documentation. Linked tables contain calculated features.
  • Figure 3: SnipGen Dataset and use. A describes the SnipGen data collection and steps until prompt generation. B describes the canonical path for training and evaluate LLMs