Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models for Multimodal Deformable Image Registration

Mingrui Ma, Weijie Wang, Jie Ning, Jianfeng He, Nicu Sebe, Bruno Lepri

TL;DR

MDIR aims to align anatomical structures across different imaging modalities, but traditional generative models can lose source information while non‑GM methods struggle with cross‑modal feature alignment. The authors propose LLM‑Morph, a coarse‑to‑fine MDIR pipeline that leverages a CNN encoder, two LLM Encoding Blocks (LEBs) with adapters, and LoRA fine‑tuning to align deep cross‑modal features and produce multi‑scale deformation fields. Visual features are progressively decoded through four adapters to generate deformation fields at multiple scales, with a Spatial Transformation Network (STN) warping enabling end‑to‑end optimization, and the method is validated on Abdomen MR‑CT and SR‑Reg Brain datasets. The study also investigates the impact of different pre‑trained layers and various LLMs to guide layer selection and model choice for MDIR, demonstrating the potential of pre‑trained LLMs to bridge modality gaps in medical image registration, with publicly available code.

Abstract

The challenge of Multimodal Deformable Image Registration (MDIR) lies in the conversion and alignment of features between images of different modalities. Generative models (GMs) cannot retain the necessary information enough from the source modality to the target one, while non-GMs struggle to align features across these two modalities. In this paper, we propose a novel coarse-to-fine MDIR framework,LLM-Morph, which is applicable to various pre-trained Large Language Models (LLMs) to solve these concerns by aligning the deep features from different modal medical images. Specifically, we first utilize a CNN encoder to extract deep visual features from cross-modal image pairs, then we use the first adapter to adjust these tokens, and use LoRA in pre-trained LLMs to fine-tune their weights, both aimed at eliminating the domain gap between the pre-trained LLMs and the MDIR task. Third, for the alignment of tokens, we utilize other four adapters to transform the LLM-encoded tokens into multi-scale visual features, generating multi-scale deformation fields and facilitating the coarse-to-fine MDIR task. Extensive experiments in MR-CT Abdomen and SR-Reg Brain datasets demonstrate the effectiveness of our framework and the potential of pre-trained LLMs for MDIR task. Our code is availabel at: https://github.com/ninjannn/LLM-Morph.

Large Language Models for Multimodal Deformable Image Registration

TL;DR

MDIR aims to align anatomical structures across different imaging modalities, but traditional generative models can lose source information while non‑GM methods struggle with cross‑modal feature alignment. The authors propose LLM‑Morph, a coarse‑to‑fine MDIR pipeline that leverages a CNN encoder, two LLM Encoding Blocks (LEBs) with adapters, and LoRA fine‑tuning to align deep cross‑modal features and produce multi‑scale deformation fields. Visual features are progressively decoded through four adapters to generate deformation fields at multiple scales, with a Spatial Transformation Network (STN) warping enabling end‑to‑end optimization, and the method is validated on Abdomen MR‑CT and SR‑Reg Brain datasets. The study also investigates the impact of different pre‑trained layers and various LLMs to guide layer selection and model choice for MDIR, demonstrating the potential of pre‑trained LLMs to bridge modality gaps in medical image registration, with publicly available code.

Abstract

The challenge of Multimodal Deformable Image Registration (MDIR) lies in the conversion and alignment of features between images of different modalities. Generative models (GMs) cannot retain the necessary information enough from the source modality to the target one, while non-GMs struggle to align features across these two modalities. In this paper, we propose a novel coarse-to-fine MDIR framework,LLM-Morph, which is applicable to various pre-trained Large Language Models (LLMs) to solve these concerns by aligning the deep features from different modal medical images. Specifically, we first utilize a CNN encoder to extract deep visual features from cross-modal image pairs, then we use the first adapter to adjust these tokens, and use LoRA in pre-trained LLMs to fine-tune their weights, both aimed at eliminating the domain gap between the pre-trained LLMs and the MDIR task. Third, for the alignment of tokens, we utilize other four adapters to transform the LLM-encoded tokens into multi-scale visual features, generating multi-scale deformation fields and facilitating the coarse-to-fine MDIR task. Extensive experiments in MR-CT Abdomen and SR-Reg Brain datasets demonstrate the effectiveness of our framework and the potential of pre-trained LLMs for MDIR task. Our code is availabel at: https://github.com/ninjannn/LLM-Morph.
Paper Structure (17 sections, 2 equations, 4 figures, 3 tables)

This paper contains 17 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning-Based Multimodal Registration Methods
  4. Large Language Models
  5. Method
  6. Architecture of LLM-Morph
  7. LLM Encoding Blocks
  8. LoRA Fine-Tuning
  9. Experiments
  10. Data Preparation
  11. Implementing Details
  12. Evaluation Metrics and Baseline Methods
  13. Experimental Results
  14. Ablation Study
  15. Ablation Results of Different Pre-trained Layers
  16. ...and 2 more sections

Figures (4)

  • Figure 1: The overall architecture of the proposed LLM-Morph (left) and the details of LLM Encoding Blocks (right).
  • Figure 2: Box plots of the alignment results of all baseline methods on (a) Abdomen MR-CT and (b) SR-Reg datasets. The horizontal axis represents different anatomical structures, and the vertical axis represents the Dice values, respectively.
  • Figure 3: Slice visualization on two datasets. The boundaries in each color represent the edges of different segmentation maps. The warped grid is utilized to observe the deformation field of the current slice, while the warped slice represents the corresponding slice from the warped moving image.
  • Figure 4: Histogram results of different layers equipped in LLM-Moprh. The horizontal axis indicates the layer ordinal, and the vertical axis represents the average Dice value. (a) Results on abdomen dataset. (b) Results on brain dataset.