RJUA-QA: A Comprehensive QA Dataset for Urology

TL;DR

RJUA-QA addresses the need for high-quality, domain-specific QA data to support clinical reasoning in urology LLMs. It introduces a Chinese urology QA dataset with 2,132 QA triads and ~25,000 records across 67 diseases, built from realistic virtual patient scenarios and expert references. The construction pipeline combines data sourcing from Shanghai Renji Hospital, preprocessing, LLM-generated QA pairs, literature context collection, and rigorous human calibration, all formatted for structured reasoning. Evaluation across multiple LLMs reveals strengths and gaps, with GPT-3.5 delivering strong overall linguistic quality (Rouge-L) while medical-specialized models perform best on diagnosis and treatment advice; RJUA-QA provides a robust benchmark for clinical reasoning and LLM deployment in healthcare, with public release for broad use.

Abstract

We introduce RJUA-QA, a novel medical dataset for question answering (QA) and reasoning with clinical evidence, contributing to bridge the gap between general large language models (LLMs) and medical-specific LLM applications. RJUA-QA is derived from realistic clinical scenarios and aims to facilitate LLMs in generating reliable diagnostic and advice. The dataset contains 2,132 curated Question-Context-Answer pairs, corresponding about 25,000 diagnostic records and clinical cases. The dataset covers 67 common urological disease categories, where the disease coverage exceeds 97.6\% of the population seeking medical services in urology. Each data instance in RJUA-QA comprises: (1) a question mirroring real patient to inquiry about clinical symptoms and medical conditions, (2) a context including comprehensive expert knowledge, serving as a reference for medical examination and diagnosis, (3) a doctor response offering the diagnostic conclusion and suggested examination guidance, (4) a diagnosed clinical disease as the recommended diagnostic outcome, and (5) clinical advice providing recommendations for medical examination. RJUA-QA is the first medical QA dataset for clinical reasoning over the patient inquiries, where expert-level knowledge and experience are required for yielding diagnostic conclusions and medical examination advice. A comprehensive evaluation is conducted to evaluate the performance of both medical-specific and general LLMs on the RJUA-QA dataset. Our data is are publicly available at \url{https://github.com/alipay/RJU_Ant_QA}.

Figures (5)

• Figure 1: Distribution of Disease Categories in the RJUA-QA Datasets.
• Figure 2: Proportional Breakdown of Urological Disease Diagnosis Categories Across QA Entries.
• Figure 3: The data construction pipeline of the RJUA-QA dataset.
• Figure 4: Source and Test Objectives of the RJUA-QA Datasets.
• Figure 5: Proportion of Primary Diagnoses in department of urology Shanghai Renji Hospital (2019-2023).