Integrating Artificial Intelligence with Human Expertise: An In-depth Analysis of ChatGPT's Capabilities in Generating Metamorphic Relations

TL;DR

This work investigates the capability of GPT models to generate and evaluate metamorphic relations (MRs) for metamorphic testing (MT) across diverse systems, addressing the oracle problem. It first contrasts GPT-3.5 and GPT-4 on MR quality using a prior framework, then introduces a refined seven-criterion evaluation scheme and applies it to nine SUTs with a custom GPT MR evaluator. Results show GPT-4 generally outperforms GPT-3.5 in correctness, clarity, and novelty, while human evaluators provide nuanced judgments on correctness and domain-specific details; the study demonstrates valuable human-AI collaboration for MT MR generation and evaluation. The findings highlight GPT-4’s potential to scale MR generation and assessment, while underscoring the enduring need for expert review to ensure precise, context-specific MR validity in complex AI-enhanced systems. Overall, the paper contributes a rigorous evaluation framework and empirical evidence for integrating AI-driven MR tooling into software testing practice, informing future research on automated MT workflows and evaluator design.

Abstract

Context: This paper provides an in-depth examination of the generation and evaluation of Metamorphic Relations (MRs) using GPT models developed by OpenAI, with a particular focus on the capabilities of GPT-4 in software testing environments. Objective: The aim is to examine the quality of MRs produced by GPT-3.5 and GPT-4 for a specific System Under Test (SUT) adopted from an earlier study, and to introduce and apply an improved set of evaluation criteria for a diverse range of SUTs. Method: The initial phase evaluates MRs generated by GPT-3.5 and GPT-4 using criteria from a prior study, followed by an application of an enhanced evaluation framework on MRs created by GPT-4 for a diverse range of nine SUTs, varying from simple programs to complex systems incorporating AI/ML components. A custom-built GPT evaluator, alongside human evaluators, assessed the MRs, enabling a direct comparison between automated and human evaluation methods. Results: The study finds that GPT-4 outperforms GPT-3.5 in generating accurate and useful MRs. With the advanced evaluation criteria, GPT-4 demonstrates a significant ability to produce high-quality MRs across a wide range of SUTs, including complex systems incorporating AI/ML components. Conclusions: GPT-4 exhibits advanced capabilities in generating MRs suitable for various applications. The research underscores the growing potential of AI in software testing, particularly in the generation and evaluation of MRs, and points towards the complementarity of human and AI skills in this domain.

Figures (6)

• Figure 1: Evaluation comparison chart of MRs for the same target function between GPT-4 and GPT-3.5
• Figure 2: Sample evaluation of new MRs with the GPT Evaluator
• Figure 3: Evaluation results of basic computational functions from both human experts and GPT
• Figure 4: Evaluation results from both human experts and GPT of complex systems with no AI embedded
• Figure 5: Evaluation results from both human experts and GPT of complex systems with AI embedded