TTTFusion: A Test-Time Training-Based Strategy for Multimodal Medical Image Fusion in Surgical Robots

TL;DR

This work tackles the challenge of real-time multimodal medical image fusion in surgical robots by introducing TTTFusion, a Test-Time Training-based approach that adapts model parameters during inference based on input data. The method employs a three-component architecture (feature extraction, a dynamic TTTFusion module with channel and spatial attention, and image reconstruction) and uses per-channel feature statistics ($bc$, $c2_c$) with $Z$-score normalization to compute modality-specific weights and biases, yielding a fused feature map $F_{fused}=W_1\u000Aandot;_{ ext{MRI}}+W_2\u000Aandot;_{ ext{CT}}+B_1+B_2$. Key results show that TTTFusion surpasses static fusion methods on MRI-CT and MRI-SPECT tasks across metrics such as PSNR, SSIM, FMI, FSIM, and EN, demonstrating enhanced edge preservation and fine-grained detail fusion suitable for real-time surgical navigation. The work presents a principled self-supervised-like adaptation during testing, enabling robust handling of modality differences without requiring additional labeled data. Overall, TTTFusion establishes a new paradigm for adaptive, real-time multimodal image fusion in clinical robotics with tangible implications for improved intraoperative visualization and decision support.

Abstract

With the increasing use of surgical robots in clinical practice, enhancing their ability to process multimodal medical images has become a key research challenge. Although traditional medical image fusion methods have made progress in improving fusion accuracy, they still face significant challenges in real-time performance, fine-grained feature extraction, and edge preservation.In this paper, we introduce TTTFusion, a Test-Time Training (TTT)-based image fusion strategy that dynamically adjusts model parameters during inference to efficiently fuse multimodal medical images. By adapting the model during the test phase, our method optimizes the parameters based on the input image data, leading to improved accuracy and better detail preservation in the fusion results.Experimental results demonstrate that TTTFusion significantly enhances the fusion quality of multimodal images compared to traditional fusion methods, particularly in fine-grained feature extraction and edge preservation. This approach not only improves image fusion accuracy but also offers a novel technical solution for real-time image processing in surgical robots.

Figures (4)

• Figure 1: The figure shows the process of using a surgical robot for assistance in brain surgery. First, the patient is properly positioned for both CT and MRI imaging. Then, the robot drills a burr hole in the brain based on the fused images and performs the surgery precisely (robot image source Han et al. han2024efficacy).
• Figure 2: The application of the multimodal medical image fusion strategy based on Test-Time Training proposed by us in surgical robots.
• Figure 3: Detailed architecture of the proposed TTTFusion, which is a dynamic feature fusion framework for multimodal medical image processing.
• Figure 4: This figure compares the original MRI, SPECT and CT images with their fused results by EH-DRAN and TTTFusion. Red boxes aid in the comparison.