ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reports

TL;DR

This work presents ReXGroundingCT, the first publicly available dataset that links free-text radiology findings to pixel-level 3D segmentations in chest CT scans. It is constructed via a three-stage pipeline: GPT-4-based report rewriting to standardize language, a two-step abnormality extraction and ontology-based categorization, and meticulous 3D annotation by radiologists complemented by an anatomical chain-of-thought resource generated with GPT-4o. Evaluation shows that off-the-shelf text-prompted 3D grounding models perform poorly, but fine-tuning on ReXGroundingCT yields meaningful gains (Global Dice ~0.21, Global HIT ~0.47), underscoring both feasibility and remaining challenges. The dataset and public leaderboard are poised to drive advances in grounded radiology report generation, explainability, and anatomy-aware AI for CT imaging.

Abstract

We introduce ReXGroundingCT, the first publicly available dataset linking free-text findings to pixel-level 3D segmentations in chest CT scans. The dataset includes 3,142 non-contrast chest CT scans paired with standardized radiology reports from CT-RATE. Construction followed a structured three-stage pipeline. First, GPT-4 was used to extract and standardize findings, descriptors, and metadata from reports originally written in Turkish and machine-translated into English. Second, GPT-4o-mini categorized each finding into a hierarchical ontology of lung and pleural abnormalities. Third, 3D annotations were produced for all CT volumes: the training set was quality-assured by board-certified radiologists, and the validation and test sets were fully annotated by board-certified radiologists. Additionally, a complementary chain-of-thought dataset was created to provide step-by-step hierarchical anatomical reasoning for localizing findings within the CT volume, using GPT-4o and localization coordinates derived from organ segmentation models. ReXGroundingCT contains 16,301 annotated entities across 8,028 text-to-3D-segmentation pairs, covering diverse radiological patterns from 3,142 non-contrast CT scans. About 79% of findings are focal abnormalities and 21% are non-focal. The dataset includes a public validation set of 50 cases and a private test set of 100 cases, both annotated by board-certified radiologists. The dataset establishes a foundation for enabling free-text finding segmentation and grounded radiology report generation in CT imaging. Model performance on the private test set is hosted on a public leaderboard at https://rexrank.ai/ReXGroundingCT. The dataset is available at https://huggingface.co/datasets/rajpurkarlab/ReXGroundingCT.

Figures (9)

• Figure 1: (A) Examples of mapping narrative free-text findings from a radiology report to precise 3D spatial locations in a chest CT scan. Five representative findings illustrate various anatomical sites and patterns of disease. Numbers indicate CT slice locations where the findings are visualized. (B) Distribution of findings per category across the dataset, grouped by typical pattern (non-focal vs. focal).
• Figure 2: Illustration of the structured pipeline for converting narrative radiology reports into standardized reports. The example shows how a free-text report is parsed into discrete findings, each assessed for abnormality and prior reference, followed by categorization into defined ontology classes. Findings that have a code that starts with "H" refer to "Heart/Vessels" findings, where "L" refer to "Lungs/Airways/Pleura" findings, and "M" refers to "Mediastinum/Hila".
• Figure 3: ReXGroundingCT dataset construction pipeline. The training set was annotated with a maximum of three representative instances per finding to manage workload, while the validation and test sets were annotated exhaustively, with all visible instances segmented to provide full ground truth. In protocol 1, all cases were reviewed and refined by board-certified radiologists, ensuring high-quality segmentations. In protocol 2, medical students received structured training and were supervised by senior radiologists, who provided corrections until the students demonstrated sufficient proficiency to annotate independently. The test and validation sets were done exclusively by board-certified radiologists
• Figure 4: (A) A 3D chest CT scan displayed within the annotation platform, showing multiple pixel-level segmentations corresponding to individual findings (e.g., P1–P5), each color-coded and labeled in the left panel. (B) Associated metadata panel, showing free-text descriptions for each annotated finding. Annotators used this interface to review CT volumes and perform labeling of abnormalities.
• Figure 5: Example of an anatomical chain-of-thought (CoT) reasoning chain. The example illustrates a coarse-to-fine reasoning chain that progressively narrows localization from the whole CT volume to the relevant organ, lobe, and adjacent structures, producing a step-by-step description aligned with the finding text. The lung and lobe boundaries are generated automatically from organ segmentation models zhang2024radgenome.