SeFENet: Robust Deep Homography Estimation via Semantic-Driven Feature Enhancement

TL;DR

SeFENet tackles robust homography estimation in harsh environments by leveraging semantic information to enhance features. It introduces a hierarchical scale-aware feature extractor, a semantic-guided constraint module, and a meta-learning training framework to harmonize semantic and structural features. The method achieves state-of-the-art robustness on synthetic VOC-based data and the large-baseline CAHomo dataset, with reductions in PME and improvements in PSNR/SSIM/NCC. This approach enhances accuracy and reliability of image alignment under rain, haze, and low light, enabling better stitching, mapping, and localization in challenging real-world conditions.

Abstract

Images captured in harsh environments often exhibit blurred details, reduced contrast, and color distortion, which hinder feature detection and matching, thereby affecting the accuracy and robustness of homography estimation. While visual enhancement can improve contrast and clarity, it may introduce visual-tolerant artifacts that obscure the structural integrity of images. Considering the resilience of semantic information against environmental interference, we propose a semantic-driven feature enhancement network for robust homography estimation, dubbed SeFENet. Concretely, we first introduce an innovative hierarchical scale-aware module to expand the receptive field by aggregating multi-scale information, thereby effectively extracting image features under diverse harsh conditions. Subsequently, we propose a semantic-guided constraint module combined with a high-level perceptual framework to achieve degradation-tolerant with semantic feature. A meta-learning-based training strategy is introduced to mitigate the disparity between semantic and structural features. By internal-external alternating optimization, the proposed network achieves implicit semantic-wise feature enhancement, thereby improving the robustness of homography estimation in adverse environments by strengthening the local feature comprehension and context information extraction. Experimental results under both normal and harsh conditions demonstrate that SeFENet significantly outperforms SOTA methods, reducing point match error by at least 41% on the large-scale datasets.

Figures (12)

• Figure 1: The concept diagram of this paper is shown above. The top half is other homography estimation methods and the bottom half is our homography estimation method. After integrating the semantics, it can be seen from the 3D topographic map that the distorted image of our homography estimation is closer to the source image than other methods in harsh environments.
• Figure 2: The overall pipeline of SeFENet. (a) is the detailed structure of target aware homography estimation module (TAHEM). (b) is the semantic extraction module (SEM). (c) is the structure of our semantic-guide meta constrains (SMC). Among the above modules, only (a) is used for inference, and the other modules are only used for the above processing during training.
• Figure 3: The implementation details of hierarchical scale-aware module. $F$ denotes the original feature. $F_\mathrm{D}$ denotes the feature after being downsampled. The pre-sampling is non-overlapping sampling and the sampling region will overlap after sampling. Same feature location is represented by same color. Multihead attention mechanism will be implemented interactively after sampling.
• Figure 4: The details of the implementation of the meta-learning strategy are divided into 2 loops, inner and outer, in which the dashed box contains the module trained by backpropagation and the rest of the content is fixed.
• Figure 5: The 4 plots respectively indicate the evaluation of our homography estimation method on the VOC dataset in 4 environments: normal, low-light, haze, and rain. The x-axis represents the average corner error (ACE) and the y-axis represents the percentage of data under the corresponding ACE.