Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection

Suavis Giramata, Madhusudan Srinivasan, Venkat Naidu Gudivada, Upulee Kanewala

TL;DR

This work tackles fairness testing in large language models by prioritizing metamorphic relations (MRs) through a sentence-diversity framework. It introduces six text-based diversity metrics (cosine similarity, lexical diversity, NER diversity, semantic similarity, sentiment similarity, and tone-based diversity) and combines them into a Final Diversity Score (FDS) to rank MRs for efficient bias detection. Empirical evaluation on GPT-4.0 and LLaMA 3.0 across 11 MRs demonstrates that the proposed prioritization yields higher fault-detection rates and faster first-fault results than random and distance-based baselines, with competitive performance relative to fault-based ordering and substantially lower computational cost. The approach offers a practical, model-agnostic tool for scalable fairness testing in LLMS, reducing testing overhead while maintaining robust bias detection across diverse MRs and domains.

Abstract

Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs. This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs. Given the exponential growth of possible test cases, exhaustive testing is impractical; therefore, prioritizing MRs based on their effectiveness in detecting fairness violations is crucial. We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection. Experimental results demonstrate that our proposed prioritization approach improves fault detection rates by 22% compared to random prioritization and 12% compared to distance-based prioritization, while reducing the time to the first failure by 15% and 8%, respectively. Furthermore, our approach performs within 5% of fault-based prioritization in effectiveness, while significantly reducing the computational cost associated with fault labeling. These results validate the effectiveness of diversity-based MR prioritization in enhancing fairness testing for LLMs.

Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection

TL;DR

This work tackles fairness testing in large language models by prioritizing metamorphic relations (MRs) through a sentence-diversity framework. It introduces six text-based diversity metrics (cosine similarity, lexical diversity, NER diversity, semantic similarity, sentiment similarity, and tone-based diversity) and combines them into a Final Diversity Score (FDS) to rank MRs for efficient bias detection. Empirical evaluation on GPT-4.0 and LLaMA 3.0 across 11 MRs demonstrates that the proposed prioritization yields higher fault-detection rates and faster first-fault results than random and distance-based baselines, with competitive performance relative to fault-based ordering and substantially lower computational cost. The approach offers a practical, model-agnostic tool for scalable fairness testing in LLMS, reducing testing overhead while maintaining robust bias detection across diverse MRs and domains.

Abstract

Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs. This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs. Given the exponential growth of possible test cases, exhaustive testing is impractical; therefore, prioritizing MRs based on their effectiveness in detecting fairness violations is crucial. We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection. Experimental results demonstrate that our proposed prioritization approach improves fault detection rates by 22% compared to random prioritization and 12% compared to distance-based prioritization, while reducing the time to the first failure by 15% and 8%, respectively. Furthermore, our approach performs within 5% of fault-based prioritization in effectiveness, while significantly reducing the computational cost associated with fault labeling. These results validate the effectiveness of diversity-based MR prioritization in enhancing fairness testing for LLMs.
Paper Structure (33 sections, 7 equations, 4 figures, 2 tables)

This paper contains 33 sections, 7 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction & Motivation
  2. Background
  3. Large Language Models (LLM)
  4. Metamorphic Testing (MT)
  5. Related Works
  6. Methodology
  7. Methodology Overview
  8. Sentence Diversity Approach
  9. Cosine Similarity based
  10. Lexical diversity based
  11. Named Entity Recognition based
  12. Semantic Similarity based
  13. Sentiment Similarity based
  14. Tone-Based Diversity
  15. Example
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Fault detection rate of MRs for GPT4.0
  • Figure 2: Time to first failure for MRs for GPT4.0
  • Figure 3: Fault detection rate of MRs for LlaMa 3
  • Figure 4: Time to first failure for MRs for LlaMa 3