Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MILE: A Mutation Testing Framework of In-Context Learning Systems

Zeming Wei, Yihao Zhang, Meng Sun

TL;DR

This work proposes a mutation testing framework designed to characterize the quality and effectiveness of test data for ICL systems, and proposes several mutation operators specialized for ICL demonstrations, as well as corresponding mutation scores for ICL test sets.

Abstract

In-context Learning (ICL) has achieved notable success in the applications of large language models (LLMs). By adding only a few input-output pairs that demonstrate a new task, the LLM can efficiently learn the task during inference without modifying the model parameters. Such mysterious ability of LLMs has attracted great research interests in understanding, formatting, and improving the in-context demonstrations, while still suffering from drawbacks like black-box mechanisms and sensitivity against the selection of examples. In this work, inspired by the foundations of adopting testing techniques in machine learning (ML) systems, we propose a mutation testing framework designed to characterize the quality and effectiveness of test data for ICL systems. First, we propose several mutation operators specialized for ICL demonstrations, as well as corresponding mutation scores for ICL test sets. With comprehensive experiments, we showcase the effectiveness of our framework in evaluating the reliability and quality of ICL test suites. Our code is available at https://github.com/weizeming/MILE.

MILE: A Mutation Testing Framework of In-Context Learning Systems

TL;DR

This work proposes a mutation testing framework designed to characterize the quality and effectiveness of test data for ICL systems, and proposes several mutation operators specialized for ICL demonstrations, as well as corresponding mutation scores for ICL test sets.

Abstract

In-context Learning (ICL) has achieved notable success in the applications of large language models (LLMs). By adding only a few input-output pairs that demonstrate a new task, the LLM can efficiently learn the task during inference without modifying the model parameters. Such mysterious ability of LLMs has attracted great research interests in understanding, formatting, and improving the in-context demonstrations, while still suffering from drawbacks like black-box mechanisms and sensitivity against the selection of examples. In this work, inspired by the foundations of adopting testing techniques in machine learning (ML) systems, we propose a mutation testing framework designed to characterize the quality and effectiveness of test data for ICL systems. First, we propose several mutation operators specialized for ICL demonstrations, as well as corresponding mutation scores for ICL test sets. With comprehensive experiments, we showcase the effectiveness of our framework in evaluating the reliability and quality of ICL test suites. Our code is available at https://github.com/weizeming/MILE.
Paper Structure (25 sections, 3 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 25 sections, 3 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. In-context learning (ICL).
  4. Mutation Testing.
  5. Mutation Testing For In-Context Learning
  6. Overview
  7. Mutation Operators for ICL
  8. Demonstration-level mutation operators.
  9. Prompt-level mutation operators.
  10. Mutation Scores
  11. Experiments
  12. Experiment Set-up
  13. Datasets.
  14. LLMs for evaluation.
  15. Mutant implementation details.
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Comparing mutation scores with different dataset sizes. Each figure represents the scores averaged over 5 datasets for a model. The X-axis denotes the ratio of the set size to $n$, and the Y-axis denotes the score (%). The blue lines represent the uni. dataset and red lines represent the non. datasets. The solid line and dotted line denote $MS_S$ and $MS_G$, respectively.
  • Figure 2: Individual $MS_G$ comparison for each mutant group on different datasets. The scores are averaged over 3 models.

Theorems & Definitions (5)

  • definition thmcounterdefinition: In-context Learning System
  • definition thmcounterdefinition: Mutation Testing
  • definition thmcounterdefinition: Standard Mutation Score
  • definition thmcounterdefinition: Group-wise Mutation Score
  • definition thmcounterdefinition: Individual Group-wise Mutation Score