Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasound

Hassan Rasheed, Reuben Dorent, Maximilian Fehrentz, Tina Kapur, William M. Wells, Alexandra Golby, Sarah Frisken, Julia A. Schnabel, Nazim Haouchine

TL;DR

A texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance images with intraoperative Ultrasound (US) images is proposed, outperforming the state-of-the-art methods and achieving 80.35% matching precision on average.

Abstract

We propose in this paper a texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance (MR) images with intraoperative Ultrasound (US) images. We introduce a matching-by-synthesis strategy, where intraoperative US images are synthesized from MR images accounting for multiple MR modalities and intraoperative US variability. We build our training set by enforcing keypoints localization over all images then train a patient-specific descriptor network that learns texture-invariant discriminant features in a supervised contrastive manner, leading to robust keypoints descriptors. Our experiments on real cases with ground truth show the effectiveness of the proposed approach, outperforming the state-of-the-art methods and achieving 80.35% matching precision on average.

Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasound

TL;DR

A texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance images with intraoperative Ultrasound (US) images is proposed, outperforming the state-of-the-art methods and achieving 80.35% matching precision on average.

Abstract

We propose in this paper a texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance (MR) images with intraoperative Ultrasound (US) images. We introduce a matching-by-synthesis strategy, where intraoperative US images are synthesized from MR images accounting for multiple MR modalities and intraoperative US variability. We build our training set by enforcing keypoints localization over all images then train a patient-specific descriptor network that learns texture-invariant discriminant features in a supervised contrastive manner, leading to robust keypoints descriptors. Our experiments on real cases with ground truth show the effectiveness of the proposed approach, outperforming the state-of-the-art methods and achieving 80.35% matching precision on average.
Paper Structure (15 sections, 2 equations, 5 figures, 2 tables)

This paper contains 15 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Overview and Problem Setting
  4. Intraoperative Image Synthesis
  5. Building the Training Dataset
  6. Learning Cross-modal Feature Descriptor
  7. Model Architecture and Loss Function
  8. Training and Optimization Details
  9. Run-time Inference and Matching
  10. Experimentations and Results
  11. Data.
  12. Metrics.
  13. Ablation study and Evaluation.
  14. State-of-the-art comparison.
  15. Conclusion

Figures (5)

  • Figure 1: Method overview: We rely on training images composed of one MR image and multiple synthesized US images, generated under different modes and noise levels (a). We train a Siamese network on image patches to learn similar and dissimilar features in a supervised contrastive manner (b). Applying this network to patches from each image leads a MR-US cross-modal matching (c).
  • Figure 2: Synthetic US image generations for three different T2 MR images (One case per row). The first column shows T2 MR; the middle columns show samples of synthetic US images generated using different combinations of T2, T1, and FLAIR with different speckles; the last column shows the ground truth US image.
  • Figure 3: Examples of matching on three cases, one per column (MR on left and US on right). From top to bottom: SIFT+Cosine, MIND+Cosine, SP+Cosine, SP+LG, Ours+LG, Ours+Cosine. Correct matches recovered by each method are shown in green lines and mismatched are shown with a red dot.
  • Figure 4: Repeatability of matches over slices (left) and textures changes (right).
  • Figure 5: MR slice #40 retrieved in US volume using descriptor matching.