Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Generating Executable Metamorphic Relations Using Large Language Models

Seung Yeob Shin, Fabrizio Pastore, Domenico Bianculli, Alexandra Baicoianu

TL;DR

This paper proposes an approach for automatically deriving executable MRs (EMRs) from requirements using large language models (LLMs), and relies on a few-shot prompting strategy to instruct the LLM to perform activities in the MT process, by providing requirements and API specifications, as one would do with software engineers.

Abstract

Metamorphic testing (MT) has proven to be a successful solution to automating testing and addressing the oracle problem. However, it entails manually deriving metamorphic relations (MRs) and converting them into an executable form; these steps are time-consuming and may prevent the adoption of MT. In this paper, we propose an approach for automatically deriving executable MRs (EMRs) from requirements using large language models (LLMs). Instead of merely asking the LLM to produce EMRs, our approach relies on a few-shot prompting strategy to instruct the LLM to perform activities in the MT process, by providing requirements and API specifications, as one would do with software engineers. To assess the feasibility of our approach, we conducted a questionnaire-based survey in collaboration with Siemens Industry Software, a worldwide leader in providing industry software and services, focusing on four of their software applications. Additionally, we evaluated the accuracy of the generated EMRs for a Web application. The outcomes of our study are highly promising, as they demonstrate the capability of our approach to generate MRs and EMRs that are both comprehensible and pertinent for testing purposes.

Towards Generating Executable Metamorphic Relations Using Large Language Models

TL;DR

This paper proposes an approach for automatically deriving executable MRs (EMRs) from requirements using large language models (LLMs), and relies on a few-shot prompting strategy to instruct the LLM to perform activities in the MT process, by providing requirements and API specifications, as one would do with software engineers.

Abstract

Metamorphic testing (MT) has proven to be a successful solution to automating testing and addressing the oracle problem. However, it entails manually deriving metamorphic relations (MRs) and converting them into an executable form; these steps are time-consuming and may prevent the adoption of MT. In this paper, we propose an approach for automatically deriving executable MRs (EMRs) from requirements using large language models (LLMs). Instead of merely asking the LLM to produce EMRs, our approach relies on a few-shot prompting strategy to instruct the LLM to perform activities in the MT process, by providing requirements and API specifications, as one would do with software engineers. To assess the feasibility of our approach, we conducted a questionnaire-based survey in collaboration with Siemens Industry Software, a worldwide leader in providing industry software and services, focusing on four of their software applications. Additionally, we evaluated the accuracy of the generated EMRs for a Web application. The outcomes of our study are highly promising, as they demonstrate the capability of our approach to generate MRs and EMRs that are both comprehensible and pertinent for testing purposes.
Paper Structure (10 sections, 4 figures, 7 tables)

This paper contains 10 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Overview
  4. Deriving metamorphic relations
  5. Generating executable metamorphic relations
  6. Experiment Design
  7. Experiment results
  8. Threats to Validity and Research Opportunities
  9. Related Work
  10. Conclusion and Future work

Figures (4)

  • Figure 1: Our approach for generating EMRs using LLMs.
  • Figure 2: Prompts to derive MRs from the SUT's requirements, grouped by phases in the conversation: (1) setting the context, (2) providing requirements specification document(s), (3) identifying relevant sentences, and (4) rewriting these sentences into MRs.
  • Figure 3: Prompts to convert MRs into EMRs, grouped by phases in the conversation: (1) setting the context, (2) learning the syntax of SMRL, (3) instructing the output format, (4) few-shot learning of SMRL, (5) learning the SUT's APIs, and (6) transforming MRs into EMRs.
  • Figure 4: An EMR generated by ChatGPT based on the MR presented in Table \ref{['tbl:req and mr']}.