RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis

TL;DR

This paper introduces RadGenome-Chest CT, a comprehensive, large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE, and believes that RadGenome-Chest CT can significantly advance the development of multimodal medical foundation models.

Abstract

Developing generalist foundation model has recently attracted tremendous attention among researchers in the field of AI for Medicine (AI4Medicine). A pivotal insight in developing these models is their reliance on dataset scaling, which emphasizes the requirements on developing open-source medical image datasets that incorporate diverse supervision signals across various imaging modalities. In this paper, we introduce RadGenome-Chest CT, a comprehensive, large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. Specifically, we leverage the latest powerful universal segmentation and large language models, to extend the original datasets (over 25,692 non-contrast 3D chest CT volume and reports from 20,000 patients) from the following aspects: (i) organ-level segmentation masks covering 197 categories, which provide intermediate reasoning visual clues for interpretation; (ii) 665 K multi-granularity grounded reports, where each sentence of the report is linked to the corresponding anatomical region of CT volume in the form of a segmentation mask; (iii) 1.3 M grounded VQA pairs, where questions and answers are all linked with reference segmentation masks, enabling models to associate visual evidence with textual explanations. All grounded reports and VQA pairs in the validation set have gone through manual verification to ensure dataset quality. We believe that RadGenome-Chest CT can significantly advance the development of multimodal medical foundation models, by training to generate texts based on given segmentation regions, which is unattainable with previous relevant datasets. We will release all segmentation masks, grounded reports, and VQA pairs to facilitate further research and development in this field.

Figures (5)

• Figure 1: Data construction pipeline of RadGenome-Chest CT.
• Figure 2: Overview of results obtained from each step of the data construction pipeline.
• Figure 3: Partial visualization of the anatomical hierarchical tree with abnormalities annotated.
• Figure 4: (a) Distribution of Normal vs. Abnormal Cases: Training set analysis. (b) Distribution of Normal vs. Abnormal Cases: Validation set analysis. (c) Abnormalities Word Cloud: Visual summary of frequent abnormalities from findings. (d) Disorders Word Cloud: Visual summary of frequent disorders from findings.
• Figure 5: Case examples of segmentation masks and structured reports.