RobustMat: Neural Diffusion for Street Landmark Patch Matching under Challenging Environments

TL;DR

RobustMat tackles robust matching of street-landmark patches under environmental perturbations by integrating a CNN-based neural ODE diffusion for per-patch feature learning with a graph neural PDE diffusion to aggregate neighborhood information. The approach constructs a neighborhood graph for each landmark patch, learns vertex and graph embeddings through diffusion processes, and combines vertex-to-vertex with vertex-to-graph similarities to produce a final matching score. Theoretical analysis establishes stability of the embeddings under perturbations, and experiments on KITTI, Oxford, and Boreas show state-of-the-art robustness to noise without relying on pre-denoising, with ablations validating the complementary roles of the neural ODE and PDE modules. This work advances patch-level landmark matching for autonomous driving by delivering robust, neighborhood-aware representations that improve place recognition, loop closure, and odometry tasks in adverse conditions.

Abstract

For autonomous vehicles (AVs), visual perception techniques based on sensors like cameras play crucial roles in information acquisition and processing. In various computer perception tasks for AVs, it may be helpful to match landmark patches taken by an onboard camera with other landmark patches captured at a different time or saved in a street scene image database. To perform matching under challenging driving environments caused by changing seasons, weather, and illumination, we utilize the spatial neighborhood information of each patch. We propose an approach, named RobustMat, which derives its robustness to perturbations from neural differential equations. A convolutional neural ODE diffusion module is used to learn the feature representation for the landmark patches. A graph neural PDE diffusion module then aggregates information from neighboring landmark patches in the street scene. Finally, feature similarity learning outputs the final matching score. Our approach is evaluated on several street scene datasets and demonstrated to achieve state-of-the-art matching results under environmental perturbations.

Figures (14)

• Figure 1: The diagram of image patch matching for street scene landmarks under a noisy environment. The landmark patches in the green bounding boxes include traffic signs, traffic lights, poles, and windows.
• Figure 2: RobustMat: robust landmark patch matching model with the neural diffusion learning. The modules for neural ODE/PDE and the discriminators are shared networks, respectively.
• Figure 3: Examples of noisy landmark patch pairs and their corresponding clean pairs from the KITTI and Oxford datasets.
• Figure 4: Examples of landmark patch pairs from the Boreas datasets.
• Figure 5: Matching prediction results for example landmark patch pairs from the noisy KITTI dataset. The prediction "1" or "0" indicates matched or unmatched. GT stands for "ground truth".

Theorems & Definitions (4)

• proof
• proof
• proof
• proof