Multi-RADS Synthetic Radiology Report Dataset and Head-to-Head Benchmarking of 41 Open-Weight and Proprietary Language Models

TL;DR

The paper tackles automated RADS assignment from radiology reports by introducing RXL-RADSet, a radiologist-verified synthetic benchmark spanning 10 RADS and 1,600 reports to evaluate structured RADS extraction. It benchmarks 41 open-weight SLMs (0.135–32B) against a proprietary model (GPT-5.2) under guided prompting, revealing a strong size-related performance curve and a notable gap for highly complex RADS tasks. Large open models (20–32B) approach proprietary-model performance under guided prompts, achieving near-100% validity and mid-to-high 70% accuracy, while thinking-enabled prompting further improves results. The work highlights the need for deterministic outputs, schema validation, human oversight, and potential hybrid pipelines to safely deploy LLMs in clinical RADS assignment, and provides a scalable benchmark for ongoing radiology NLP evaluation.

Abstract

Background: Reporting and Data Systems (RADS) standardize radiology risk communication but automated RADS assignment from narrative reports is challenging because of guideline complexity, output-format constraints, and limited benchmarking across RADS frameworks and model sizes. Purpose: To create RXL-RADSet, a radiologist-verified synthetic multi-RADS benchmark, and compare validity and accuracy of open-weight small language models (SLMs) with a proprietary model for RADS assignment. Materials and Methods: RXL-RADSet contains 1,600 synthetic radiology reports across 10 RADS (BI-RADS, CAD-RADS, GB-RADS, LI-RADS, Lung-RADS, NI-RADS, O-RADS, PI-RADS, TI-RADS, VI-RADS) and multiple modalities. Reports were generated by LLMs using scenario plans and simulated radiologist styles and underwent two-stage radiologist verification. We evaluated 41 quantized SLMs (12 families, 0.135-32B parameters) and GPT-5.2 under a fixed guided prompt. Primary endpoints were validity and accuracy; a secondary analysis compared guided versus zero-shot prompting. Results: Under guided prompting GPT-5.2 achieved 99.8% validity and 81.1% accuracy (1,600 predictions). Pooled SLMs (65,600 predictions) achieved 96.8% validity and 61.1% accuracy; top SLMs in the 20-32B range reached ~99% validity and mid-to-high 70% accuracy. Performance scaled with model size (inflection between <1B and >=10B) and declined with RADS complexity primarily due to classification difficulty rather than invalid outputs. Guided prompting improved validity (99.2% vs 96.7%) and accuracy (78.5% vs 69.6%) compared with zero-shot. Conclusion: RXL-RADSet provides a radiologist-verified multi-RADS benchmark; large SLMs (20-32B) can approach proprietary-model performance under guided prompting, but gaps remain for higher-complexity schemes.

Figures (6)

• Figure 1: Overview of the RXL-RADSet Benchmarking Framework.
• Figure 2: Accuracy and validity as a function of model size (fig2). Each point is a model; color = family; marker = reasoning mode.
• Figure 3: Accuracy versus model size stratified by RADS complexity (fig3).
• Figure 4: Model-wise accuracy and validity across evaluated systems (fig4).
• Figure 5: Best-per-RADS accuracy by prompting (fig5).