Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Testing for LLM-Based Applications: A Diversity-based Approach

Juyeon Yoon, Robert Feldt, Shin Yoo

TL;DR

It is shown that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates.

Abstract

The recent surge of building software systems powered by Large Language Models (LLMs) has led to the development of various testing frameworks, primarily focused on treating prompt templates as the unit of testing. Despite the significant costs associated with test input execution and output assessment, the curation of optimized test suites is yet overlooked in these tools, which calls for tailored test selection or prioritization strategies. In this paper, we show that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates. Our proposed adaptive testing approach adjusts the conventional ART process to this context by selecting new test inputs based on scores derived from existing test suite and their labelling results. Our results, obtained using various implementations that explore several string-based distances, confirm that our approach enables the discovery of failures with reduced testing budgets and promotes the generation of more varied outputs.

Adaptive Testing for LLM-Based Applications: A Diversity-based Approach

TL;DR

It is shown that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates.

Abstract

The recent surge of building software systems powered by Large Language Models (LLMs) has led to the development of various testing frameworks, primarily focused on treating prompt templates as the unit of testing. Despite the significant costs associated with test input execution and output assessment, the curation of optimized test suites is yet overlooked in these tools, which calls for tailored test selection or prioritization strategies. In this paper, we show that diversity-based testing techniques, such as Adaptive Random Testing (ART) with appropriate string distance metrics, can be effectively applied to the testing of prompt templates. Our proposed adaptive testing approach adjusts the conventional ART process to this context by selecting new test inputs based on scores derived from existing test suite and their labelling results. Our results, obtained using various implementations that explore several string-based distances, confirm that our approach enables the discovery of failures with reduced testing budgets and promotes the generation of more varied outputs.
Paper Structure (26 sections, 3 equations, 7 figures, 1 table, 2 algorithms)

This paper contains 26 sections, 3 equations, 7 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. LLM-based Applications and Testing in Practice
  4. Diversity-based Testing
  5. Approach
  6. Adaptive test selection method for prompt templates
  7. Diversity-based score calculation w/ selective reference set
  8. Experimental Setup
  9. Research Questions
  10. RQ1. Failure Discovery
  11. RQ2. Output Diversity
  12. RQ3. Cost Analysis
  13. Dataset
  14. Models and Metrics
  15. Baseline and Comparison Targets
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Adaptive testing framework for prompt templates.
  • Figure 2: Percentage of found failures by varying selection percentages with different selection methods.
  • Figure 3: Failure detection by the number of executed test inputs in test prioritization scenario.
  • Figure 4: Distribution of APFD values with and without selective reference set selection.
  • Figure 5: Example failing inputs contained from the 'dbpedia' task in P3 dataset.
  • ...and 2 more figures