Testing Question Answering Software with Context-Driven Question Generation

TL;DR

QA systems risk incorrect answers, and existing metamorphic-test approaches struggle with generating natural, context-rich test questions. CQ^2A introduces a context-driven framework that extracts ground-truth entity and relationship information from the full context, then uses LLMs to generate questions anchored to those answers. It enhances reliability via constraint checking and consistency verification, plus a two-stage similarity mechanism combining embedding models and LLM judgments. Extensive experiments on BoolQ, SQuAD2, and NarrativeQA with UnifiedQA-v2-large show CQ^2A yields higher bug-detection rates, more natural and context-covering questions, and tangible improvements to QA performance when used for fine-tuning.

Abstract

Question-answering software is becoming increasingly integrated into our daily lives, with prominent examples including Apple Siri and Amazon Alexa. Ensuring the quality of such systems is critical, as incorrect answers could lead to significant harm. Current state-of-the-art testing approaches apply metamorphic relations to existing test datasets, generating test questions based on these relations. However, these methods have two key limitations. First, they often produce unnatural questions that humans are unlikely to ask, reducing the effectiveness of the generated questions in identifying bugs that might occur in real-world scenarios. Second, these questions are generated from pre-existing test datasets, ignoring the broader context and thus limiting the diversity and relevance of the generated questions. In this work, we introduce CQ^2A, a context-driven question generation approach for testing question-answering systems. Specifically, CQ^2A extracts entities and relationships from the context to form ground truth answers, and utilizes large language models to generate questions based on these ground truth answers and the surrounding context. We also propose the consistency verification and constraint checking to increase the reliability of LLM's outputs. Experiments conducted on three datasets demonstrate that CQ^2A outperforms state-of-the-art approaches on the bug detection capability, the naturalness of the generated questions as well as the coverage of the context. Moreover, the test cases generated by CQ^2A reduce error rate when utilized for fine-tuning the QA software under test

Figures (2)

• Figure 1: The deficiencies of questions generated by QAQA.
• Figure 2: The overview of $CQ^2A$