SSHNet: Unsupervised Cross-modal Homography Estimation via Problem Reformulation and Split Optimization

TL;DR

SSHNet tackles unsupervised cross-modal homography estimation by reformulating the problem into two directly supervised sub-tasks and solving them via split optimization, which stabilizes training and enables compatibility with iterative backbones like IHN and RHWF. It introduces an extra homography feature-space supervision for correlation-based decoders and a distillation regimen to reduce model size while boosting cross-domain generalization. Empirical results show SSHNet achieving state-of-the-art performance among unsupervised methods and strong results against supervised baselines across GoogleMap, DPDN, OPT-SAR, Flash/no-flash, and RGB/NIR datasets, with notable MACEs reductions on challenging tasks. The approach offers practical impact by enabling robust cross-modal registration under large modality gaps and deformations, while maintaining efficient, transferable models for real-world multi-sensor scenarios.

Abstract

We propose a novel unsupervised cross-modal homography estimation learning framework, named Split Supervised Homography estimation Network (SSHNet). SSHNet reformulates the unsupervised cross-modal homography estimation into two supervised sub-problems, each addressed by its specialized network: a homography estimation network and a modality transfer network. To realize stable training, we introduce an effective split optimization strategy to train each network separately within its respective sub-problem. We also formulate an extra homography feature space supervision to enhance feature consistency, further boosting the estimation accuracy. Moreover, we employ a simple yet effective distillation training technique to reduce model parameters and improve cross-domain generalization ability while maintaining comparable performance. The training stability of SSHNet enables its cooperation with various homography estimation architectures. Experiments reveal that the SSHNet using IHN as homography estimation network, namely SSHNet-IHN, outperforms previous unsupervised approaches by a significant margin. Even compared to supervised approaches MHN and LocalTrans, SSHNet-IHN achieves 47.4% and 85.8% mean average corner errors (MACEs) reduction on the challenging OPT-SAR dataset.

Figures (7)

• Figure 1: Comparison of the solution space between (a) Content consistency indirect supervision and (b) Homography direct supervision. The content consistency indirect supervision may mislead the training due to local minima, while homography direct supervision won't.
• Figure 2: Reformulating the unsupervised cross-modal homography estimation problem into two sub-problems with direct supervision. (a) Unsupervised cross-modal homography estimation problem without direct supervision. (b) Mono-modal homography estimation problem with imperfect inputs. (c) Modality transfer problem with imperfect supervision.
• Figure 3: Comparison of straight optimization and split optimization with regularization. (a) Cross-modal test MACEs for the network using straight optimization and split optimization with regularization, on OPT-SAR dataset. (b) Schematic diagram of straight optimization and split optimization.
• Figure 4: Schematic diagram of Split Supervised Homography estimation Network, namely SSHNet. (a) Detailed optimization process of Sub-problem I. (b) Detailed optimization process of Sub-problem II. (c) Extra supervision on the homography feature space.
• Figure 5: Schematic diagram of the distillation training.