LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine

TL;DR

This work introduces LLMeDiff, an LLM-based test-generation and differential-testing framework for a medical rule engine (GURI) used in cancer data processing. By prompting four diverse LLMs to produce Pass, Fail, and NotApplied tests for 58 real medical rules and comparing outputs to a reference engine (Dvare++), the study evaluates effectiveness, efficiency, and robustness while revealing implementation mismatches. GPT-3.5 emerges as the most effective tester with the fewest hallucinations, though it is the slowest, whereas Mixtral and Mistral are faster but slightly less accurate. The differential-testing results identify 22 inconsistent rules across versions, date formats, and variable handling, underscoring the need for domain-aware test-generation and careful root-cause analysis. The findings offer practical guidance for applying LLMs to rule-based QA in real-world health registries and outline directions for improving automated testing in evolving medical-rule systems.

Abstract

The Cancer Registry of Norway (CRN) uses an automated cancer registration support system (CaReSS) to support core cancer registry activities, i.e, data capture, data curation, and producing data products and statistics for various stakeholders. GURI is a core component of CaReSS, which is responsible for validating incoming data with medical rules. Such medical rules are manually implemented by medical experts based on medical standards, regulations, and research. Since large language models (LLMs) have been trained on a large amount of public information, including these documents, they can be employed to generate tests for GURI. Thus, we propose an LLM-based test generation and differential testing approach (LLMeDiff) to test GURI. We experimented with four different LLMs, two medical rule engine implementations, and 58 real medical rules to investigate the hallucination, success, time efficiency, and robustness of the LLMs to generate tests, and these tests' ability to find potential issues in GURI. Our results showed that GPT-3.5 hallucinates the least, is the most successful, and is generally the most robust; however, it has the worst time efficiency. Our differential testing revealed 22 medical rules where implementation inconsistencies were discovered (e.g., regarding handling rule versions). Finally, we provide insights for practitioners and researchers based on the results.

Figures (6)

• Figure 1: An example medical rule
• Figure 2: LLMeDiff Overview
• Figure 3: Completion rates for all the rules per .
• Figure 4: Success indices for all the rules per test type and .
• Figure 5: Inference times for all the rules per .