DINOv3-Guided Cross Fusion Framework for Semantic-aware CT generation from MRI and CBCT

TL;DR

This work addresses the challenge of generating semantically consistent synthetic CTs from CBCT or MRI by balancing global semantic understanding with local anatomical fidelity. It introduces the DINOv3-Guided Cross Fusion Framework (DGCF), which fuses a frozen DINOv3 Transformer encoder with a trainable CNN encoder–decoder through a learnable cross-fusion module, complemented by a Multi-Level DINOv3 Perceptual (MLDP) loss that enforces semantic alignment in the DINOv3 feature space. Experiments on the SynthRAD2023 pelvic dataset show state-of-the-art performance for MRI→CT and CBCT→CT translations in MS-SSIM, PSNR, and SegScore, and an ablation study confirms the benefit of cross fusion and DINOv3-based perceptual supervision. This approach highlights the potential of foundation-model guidance for dense medical image synthesis, enabling more accurate anatomical representation for radiation therapy planning, with code available online.

Abstract

Generating synthetic CT images from CBCT or MRI has a potential for efficient radiation dose planning and adaptive radiotherapy. However, existing CNN-based models lack global semantic understanding, while Transformers often overfit small medical datasets due to high model capacity and weak inductive bias. To address these limitations, we propose a DINOv3-Guided Cross Fusion (DGCF) framework that integrates a frozen self-supervised DINOv3 Transformer with a trainable CNN encoder-decoder. It hierarchically fuses global representation of Transformer and local features of CNN via a learnable cross fusion module, achieving balanced local appearance and contextual representation. Furthermore, we introduce a Multi-Level DINOv3 Perceptual (MLDP) loss that encourages semantic similarity between synthetic CT and the ground truth in DINOv3's feature space. Experiments on the SynthRAD2023 pelvic dataset demonstrate that DGCF achieved state-of-the-art performance in terms of MS-SSIM, PSNR and segmentation-based metrics on both MRI$\rightarrow$CT and CBCT$\rightarrow$CT translation tasks. To the best of our knowledge, this is the first work to employ DINOv3 representations for medical image translation, highlighting the potential of self-supervised Transformer guidance for semantic-aware CT synthesis. The code is available at https://github.com/HiLab-git/DGCF.

Figures (2)

• Figure 1: Overview of our DINOv3-Guided Cross Fusion Framework (DGCF) for synthetic CT generation.
• Figure 2: Visual comparison of CT synthesis quality. For each case, the last row shows segmentation obtained by TotalSegmentator from the synthesized images.