Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model

TL;DR

Text2CT tackles the challenge of generating high-resolution 3D CT volumes from free-text clinical descriptions by introducing a modular 3D compression–diffusion framework. It combines a 3D Compression Network (to reduce memory while preserving anatomy) with a text-conditional Latent Diffusion Model, guided by LLM-driven prompt formulation that translates radiology reports into diverse prompts. The approach demonstrates superior image quality and text–image alignment (lower FID, higher CLIP) on CT-RATE and RadChestCT, with clear benefits for data augmentation and clinical applicability. While computationally intensive, Text2CT establishes a practical path toward flexible, anatomically accurate 3D medical image synthesis from unstructured text, enabling better diagnostics and research tools.

Abstract

Generating 3D CT volumes from descriptive free-text inputs presents a transformative opportunity in diagnostics and research. In this paper, we introduce Text2CT, a novel approach for synthesizing 3D CT volumes from textual descriptions using the diffusion model. Unlike previous methods that rely on fixed-format text input, Text2CT employs a novel prompt formulation that enables generation from diverse, free-text descriptions. The proposed framework encodes medical text into latent representations and decodes them into high-resolution 3D CT scans, effectively bridging the gap between semantic text inputs and detailed volumetric representations in a unified 3D framework. Our method demonstrates superior performance in preserving anatomical fidelity and capturing intricate structures as described in the input text. Extensive evaluations show that our approach achieves state-of-the-art results, offering promising potential applications in diagnostics, and data augmentation.

Figures (8)

• Figure 1: Generated 3D CT volume by the proposed Text2CT (left) and GenerateCT hamamci2023generatect (right). We show the axial and sagittal views from top to bottom, respectively. 3$\times$ zoomed-in images are presented to emphasize the differences, highlighting the 3D continuity issue of using a 2D super-resolution network in GenerateCT hamamci2023generatect and the grid-like artifacts from 3D super-resolution network in MedSyn xu2024medsyn. White arrows point to areas of abnormal human anatomy and artifacts from super-resolution in hamamci2023generatectxu2024medsyn, revealing potential inaccuracies in the generated output.
• Figure 2: Top row: t-SNE plot comparing latent feature distributions from free-format and fixed-format texts. The broader spread of free-format features highlights greater diversity and variability, contrasting with the more constrained distribution of fixed-format text. Bottom row: Two examples describing the same content using free-format and fixed-format text prompts.
• Figure 3: (a) The schematics of generation pipelines for Text2CT. We employ LLM to generate the general description and augmented variants of demographics, findings, and impressions based on existing radiology reports and the list of organs derived from segmentation maps. (b) The overview of the training stage of text-conditional LDM in Text2CT.
• Figure 4: The characteristics of the CT-RATE dataset hamamci2024foundation utilized by the proposed Text2CT are detailed through the abnormality distribution (left) and the word cloud of the impression (right).
• Figure 5: Qualitative assessment of model generalizability using free-format text prompts. Abnormalities mentioned in the prompts are highlighted in color and outlined by boxes in generated images. White arrows indicate areas of abnormal anatomy and artifacts from super-resolution in hamamci2023generatectxu2024medsyn. Note the left-right reversal: the patient's right side appears on the left side of the image, and vice versa.