MedVAR: Towards Scalable and Efficient Medical Image Generation via Next-scale Autoregressive Prediction

TL;DR

MedVAR presents a scalable next-scale autoregressive framework for medical image synthesis that learns a unified, hierarchical representation across heterogeneous CT and MRI datasets. By pairing a domain-specific multi-scale VQVAE with conditioned next-scale autoregression, it achieves fast, coherent high-resolution generation while preserving fine radiological details. Across extensive experiments, MedVAR demonstrates strong fidelity, diversity, and cross-domain robustness, outperforming GANs and diffusion-based baselines in both quality and efficiency. The approach establishes a practical foundation for scalable medical generative models and enables future controllable generation with richer conditioning signals.

Abstract

Medical image generation is pivotal in applications like data augmentation for low-resource clinical tasks and privacy-preserving data sharing. However, developing a scalable generative backbone for medical imaging requires architectural efficiency, sufficient multi-organ data, and principled evaluation, yet current approaches leave these aspects unresolved. Therefore, we introduce MedVAR, the first autoregressive-based foundation model that adopts the next-scale prediction paradigm to enable fast and scale-up-friendly medical image synthesis. MedVAR generates images in a coarse-to-fine manner and produces structured multi-scale representations suitable for downstream use. To support hierarchical generation, we curate a harmonized dataset of around 440,000 CT and MRI images spanning six anatomical regions. Comprehensive experiments across fidelity, diversity, and scalability show that MedVAR achieves state-of-the-art generative performance and offers a promising architectural direction for future medical generative foundation models.

Figures (7)

• Figure 2: Summary of datasets used, including voxel counts and anatomical region–modality. * indicates in-house dataset. Voxel statistics: CT: 3146, MRI: 5179.
• Figure 3: Overview of the MedVAR framework. Raw CT/MRI data are first normalized and converted into unified 2D slices. A multi-scale VQVAE encodes images into hierarchical latent tokens, on which MedVAR performs next-scale autoregressive prediction with optional conditional dropout and CFG. During inference, multi-scale tokens are generated autoregressively and decoded to synthesize full-resolution medical images, which are evaluated in terms of fidelity, diversity, and scalability.
• Figure 4: Comparison of VQVAE codebook usage heatmaps.(a) Feeding medical images into an ImageNet-pretrained VQVAE results in extremely sparse activation (codebook collapse). (b) Natural images effectively utilize the ImageNet-pretrained codebook. (c) Our domain-specific Medical VQVAE restores dense and effective codebook utilization, capturing rich anatomical features.
• Figure 5: Model comparison on $256 \times 256$ benchmark. Each model series has been selected based on its best FID performance for comparison.
• Figure 6: Some generated $256 \times 256$ by MedVAR-d30. Zoom in for a better view. From top left to right down, it shows the diverse generation results of MedVAR-d30. They are abdomen CT, abdomen MRI, brain MRI, chest CT, heart CT, heart MRI, prostate MRI, spine CT, and spine MRI results.