Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Sensor Matching with HyperNetworks

Eli Passov, Nathan S. Netanyahu, Yosi Keller

TL;DR

Hyp-Net presents a lightweight hypernetwork-augmented Siamese CNN for multimodal (VIS–IR) patch matching. By applying Conditional Instance Normalization in shallow layers and per-channel scaling/shifting via hypernetworks in deeper layers, the model learns robust, context-aware descriptors while maintaining descriptor-based efficiency. Trained with a triplet margin loss and hard negative mining, it achieves state-of-the-art results on VIS-NIR and VIS-LWIR benchmarks and competitive performance on others, with strong cross-domain generalization demonstrated on the GAP-VIR dataset. The GAP-VIR dataset further provides a challenging benchmark for cross-platform VIS–IR adaptation, encouraging future advances in cross-modal patch correspondence and domain transfer.

Abstract

Hypernetworks are models that generate or modulate the weights of another network. They provide a flexible mechanism for injecting context and task conditioning and have proven broadly useful across diverse applications without significant increases in model size. We leverage hypernetworks to improve multimodal patch matching by introducing a lightweight descriptor-learning architecture that augments a Siamese CNN with (i) hypernetwork modules that compute adaptive, per-channel scaling and shifting and (ii) conditional instance normalization that provides modality-specific adaptation (e.g., visible vs. infrared, VIS-IR) in shallow layers. This combination preserves the efficiency of descriptor-based methods during inference while increasing robustness to appearance shifts. Trained with a triplet loss and hard-negative mining, our approach achieves state-of-the-art results on VIS-NIR and other VIS-IR benchmarks and matches or surpasses prior methods on additional datasets, despite their higher inference cost. To spur progress on domain shift, we also release GAP-VIR, a cross-platform (ground/aerial) VIS-IR patch dataset with 500K pairs, enabling rigorous evaluation of cross-domain generalization and adaptation.

Multi-Sensor Matching with HyperNetworks

TL;DR

Hyp-Net presents a lightweight hypernetwork-augmented Siamese CNN for multimodal (VIS–IR) patch matching. By applying Conditional Instance Normalization in shallow layers and per-channel scaling/shifting via hypernetworks in deeper layers, the model learns robust, context-aware descriptors while maintaining descriptor-based efficiency. Trained with a triplet margin loss and hard negative mining, it achieves state-of-the-art results on VIS-NIR and VIS-LWIR benchmarks and competitive performance on others, with strong cross-domain generalization demonstrated on the GAP-VIR dataset. The GAP-VIR dataset further provides a challenging benchmark for cross-platform VIS–IR adaptation, encouraging future advances in cross-modal patch correspondence and domain transfer.

Abstract

Hypernetworks are models that generate or modulate the weights of another network. They provide a flexible mechanism for injecting context and task conditioning and have proven broadly useful across diverse applications without significant increases in model size. We leverage hypernetworks to improve multimodal patch matching by introducing a lightweight descriptor-learning architecture that augments a Siamese CNN with (i) hypernetwork modules that compute adaptive, per-channel scaling and shifting and (ii) conditional instance normalization that provides modality-specific adaptation (e.g., visible vs. infrared, VIS-IR) in shallow layers. This combination preserves the efficiency of descriptor-based methods during inference while increasing robustness to appearance shifts. Trained with a triplet loss and hard-negative mining, our approach achieves state-of-the-art results on VIS-NIR and other VIS-IR benchmarks and matches or surpasses prior methods on additional datasets, despite their higher inference cost. To spur progress on domain shift, we also release GAP-VIR, a cross-platform (ground/aerial) VIS-IR patch dataset with 500K pairs, enabling rigorous evaluation of cross-domain generalization and adaptation.
Paper Structure (17 sections, 2 equations, 3 figures, 10 tables, 1 algorithm)

This paper contains 17 sections, 2 equations, 3 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hypernetworks
  4. Descriptor Learning vs. Similarity-Based Methods for Patch Matching
  5. Patch Matching using Hyper-Networks
  6. Network Architecture
  7. Hypernetwork Module
  8. Loss Function
  9. GAP-VIR Dataset
  10. Experimental Results
  11. Datasets
  12. Implementation Details
  13. Results
  14. Ablation studies
  15. Conclusion
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Top: Patch matching between visible (grayscale) and NIR images. Bottom: The proposed hypernetwork-based Siamese CNN, where each branch encodes patches into descriptors. Most weights are shared; deeper layers include convolutions, hypernetwork modules, and batch normalization (BN), while shallow layers use convolutions and conditional instance normalization (CIN) for modality-specific adaptation. A symmetric triplet loss trains the network to learn discriminative descriptors.
  • Figure 2: Hypernetwork module. The architecture of the hypernetwork: the convolution input is passed through global pooling, followed by fully connected nonlinear layers that produce the scaling and shifting factors. These factors are applied to the convolution output after being replicated across spatial dimensions.
  • Figure 3: Patch extraction from paired aerial (a) and ground (b) imagery. Top: FAST keypoints (red) and selected keypoints (blue) overlaid on IR (left) and RGB (right) images. Bottom: corresponding $64\times64$ patches from both modalities. The detailed patch extraction procedure is provided in Supplementary Appendix A.