Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation

TL;DR

This work targets 3D radiology report generation by creating CT-3DRRG, the largest public benchmark for 3D CT-to-text tasks, and introducing Argus, a scalable vision-language framework optimized for high-resolution 3D CT inputs. It provides a comprehensive analysis of VLM design choices, including 3D vision encoder pretraining, visual token compression (favoring pixel shuffle), and a two-stage training schedule with unfrozen ViT, demonstrating robust gains across normal and high-resolution CT volumes up to $512\times512\times256$. The CT-3DRRG benchmark combines data from BIMCV-R, CT-RATE, and INSPECT and employs both NLP and clinical efficacy metrics (e.g., GREEN, RaTEScore, RadGraphXL) to ensure clinically meaningful evaluation across sources. The Argus family (3B–70B) achieves state-of-the-art performance on 3DRRG, with smaller models outperforming larger baselines and demonstrated improvements through data/model scaling, highlighting practical potential for economical, high-quality automated radiology reporting in clinical workflows.

Abstract

Automatic radiology report generation holds significant potential to streamline the labor-intensive process of report writing by radiologists, particularly for 3D radiographs such as CT scans. While CT scans are critical for clinical diagnostics, they remain less explored compared to 2D radiographs. To date, there has been no comprehensive benchmark for 3D radiograph report generation (3DRRG), nor sufficient investigation into the optimal training strategies for Vision Language Models (VLMs) in this context, particularly with respect to vision encoder choices, visual token compression, and model scaling. In this work, we make three key contributions. We curate **CT-3DRRG**, the largest **publicly** available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG. Furthermore, we propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data & model scale. Guided by these findings, we introduce **Argus**, a state-of-the-art family of VLMs that achieve superior performance across different model sizes and input 3D medical image resolutions, efficiently processing high-resolution 3D images up to $512 \times 512 \times 256$[^1].

Figures (7)

• Figure 1: Argus framework. A schematic illustration of our comprehensive exploration of 3DRRG, covering key design choices from 3D vision encoder pretraining and visual token compression to training schedules and scalability. We systematically analyze each component in Sections \ref{['sec:vlm design']} and \ref{['sec:train schedule']}.
• Figure 2: Distribution of the CT-3DRRG dataset. Left: Word cloud visualization highlighting the most frequent terms in the radiology reports. Right: Histogram showing the distribution of report lengths (measured in number of tokens).
• Figure 3: Comparison of different model combinations involving various vision encoder pretraining strategies, visual token compression, and connector designs under (a) normal resolution and (b) high resolution settings. The results represent the average clinical metric across the three subsets mentioned in Section \ref{['sec:dataset']}.
• Figure 4: Comparison of different training schedules, including multi-stage strategies and variations of ViT, under (a) normal resolution and (b) high resolution settings. The results reflect the average clinical metric computed across the three subsets outlined in Section \ref{['sec:dataset']}.
• Figure 5: Ablation study on the masking ratio for vision encoder pretraining. We evaluate the impact of the mask ratio with 3D-MAE combined with the FLIP strategy using a 8B LLM.