Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HMAR: Hierarchical Modality-Aware Expert and Dynamic Routing Medical Image Retrieval Architecture

Aojie Yuan

Abstract

Medical image retrieval (MIR) is a critical component of computer-aided diagnosis, yet existing systems suffer from three persistent limitations: uniform feature encoding that fails to account for the varying clinical importance of anatomical structures, ambiguous similarity metrics based on coarse classification labels, and an exclusive focus on global image similarity that cannot meet the clinical demand for fine-grained region-specific retrieval. We propose HMAR (Hierarchical Modality-Aware Expert and Dynamic Routing), an adaptive retrieval framework built on a Mixture-of-Experts (MoE) architecture. HMAR employs a dual-expert mechanism: Expert0 extracts global features for holistic similarity matching, while Expert1 learns position-invariant local representations for precise lesion-region retrieval. A two-stage contrastive learning strategy eliminates the need for expensive bounding-box annotations, and a sliding-window matching algorithm enables dense local comparison at inference time. Hash codes are generated via Kolmogorov-Arnold Network (KAN) layers for efficient Hamming-distance search. Experiments on the RadioImageNet-CT dataset (16 clinical patterns, 29,903 images) show that HMAR achieves mean Average Precision (mAP) of 0.711 and 0.724 for 64-bit and 128-bit hash codes, improving over the state-of-the-art ACIR method by 0.7% and 1.1%, respectively.

HMAR: Hierarchical Modality-Aware Expert and Dynamic Routing Medical Image Retrieval Architecture

Abstract

Medical image retrieval (MIR) is a critical component of computer-aided diagnosis, yet existing systems suffer from three persistent limitations: uniform feature encoding that fails to account for the varying clinical importance of anatomical structures, ambiguous similarity metrics based on coarse classification labels, and an exclusive focus on global image similarity that cannot meet the clinical demand for fine-grained region-specific retrieval. We propose HMAR (Hierarchical Modality-Aware Expert and Dynamic Routing), an adaptive retrieval framework built on a Mixture-of-Experts (MoE) architecture. HMAR employs a dual-expert mechanism: Expert0 extracts global features for holistic similarity matching, while Expert1 learns position-invariant local representations for precise lesion-region retrieval. A two-stage contrastive learning strategy eliminates the need for expensive bounding-box annotations, and a sliding-window matching algorithm enables dense local comparison at inference time. Hash codes are generated via Kolmogorov-Arnold Network (KAN) layers for efficient Hamming-distance search. Experiments on the RadioImageNet-CT dataset (16 clinical patterns, 29,903 images) show that HMAR achieves mean Average Precision (mAP) of 0.711 and 0.724 for 64-bit and 128-bit hash codes, improving over the state-of-the-art ACIR method by 0.7% and 1.1%, respectively.
Paper Structure (30 sections, 25 equations, 7 figures, 1 table)

This paper contains 30 sections, 25 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Uniform feature encoding.
  3. Ambiguous similarity metrics.
  4. Single retrieval granularity.
  5. Related Work
  6. Deep Hashing for Image Retrieval
  7. Medical Image Retrieval
  8. Mixture of Experts
  9. Contrastive Learning
  10. Method
  11. System Overview
  12. Feature Extraction Backbone
  13. Mixture-of-Experts Gating
  14. KAN-Based Hash Encoding
  15. Sliding-Window Matching
  16. ...and 15 more sections

Figures (7)

  • Figure 1: Overall architecture of HMAR. Expert$_0$ handles global feature extraction; Expert$_1$ specializes in position-invariant local features. The gating function dynamically routes between global and local retrieval based on user input.
  • Figure 2: Sliding-window matching mechanism. The query bounding box is mapped to feature-map coordinates, and a same-sized window scans each candidate image to find the region with minimum Hamming distance.
  • Figure 3: Training objective of Stage 2. Geometric augmentations (rotation, flipping) are applied to the same image to create positive pairs, encouraging Expert$_1$ to learn features invariant to spatial transformations.
  • Figure 4: Performance comparison between HMAR and ACIR across different hash-bit settings on RadIN-CT.
  • Figure 5: Global retrieval example. The query image (left) and top retrieved candidates (right) exhibit similar overall anatomical and pathological characteristics.
  • ...and 2 more figures