JamMa: Ultra-lightweight Local Feature Matching with Joint Mamba

TL;DR

JamMa addresses the efficiency–performance gap in local feature matching by extending Mamba to joint cross-view matching with the JEGO scan-merge strategy, achieving linear complexity $O(N)$ on a single GPU. The method employs a four-directional JEGO scanning strategy and a gating aggregator to produce global omnidirectional representations, enabling high-frequency mutual interaction while maintaining low parameter and FLOP counts. A two-stage coarse-to-fine pipeline uses an MLP-Mixer for fine matching and a sub-pixel regression step, trained with a trio of losses anchored in camera geometry. Empirical results on MegaDepth and HPatches show JamMa delivers an excellent performance–efficiency balance among sparse/semi-dense methods, with competitive visual localization performance and far fewer parameters than transformer-based counterparts.

Abstract

Existing state-of-the-art feature matchers capture long-range dependencies with Transformers but are hindered by high spatial complexity, leading to demanding training and highlatency inference. Striking a better balance between performance and efficiency remains a challenge in feature matching. Inspired by the linear complexity O(N) of Mamba, we propose an ultra-lightweight Mamba-based matcher, named JamMa, which converges on a single GPU and achieves an impressive performance-efficiency balance in inference. To unlock the potential of Mamba for feature matching, we propose Joint Mamba with a scan-merge strategy named JEGO, which enables: (1) Joint scan of two images to achieve high-frequency mutual interaction, (2) Efficient scan with skip steps to reduce sequence length, (3) Global receptive field, and (4) Omnidirectional feature representation. With the above properties, the JEGO strategy significantly outperforms the scan-merge strategies proposed in VMamba and EVMamba in the feature matching task. Compared to attention-based sparse and semi-dense matchers, JamMa demonstrates a superior balance between performance and efficiency, delivering better performance with less than 50% of the parameters and FLOPs.

Figures (20)

• Figure 1: Efficiency vs. Performance. State-of-the-art sparse and semi-dense methods are compared in the MegaDepth dataset megadepth. In all three commonly used efficiency metrics, the proposed JamMa achieves a superior performance-efficiency balance by a clear margin.
• Figure 2: Receptive Fields and Sequence Directions of Visual Mamba Models.
• Figure 3: Overview of the Proposed Method. JamMa extracts coarse and fine local features with a CNN encoder (\ref{['subsec:encoder']}) and scans the coarse features with the JEGO scan module (\ref{['subsec:scan']}). The four sequences are processed by four independent Mamba blocks and then merged back into 2D feature maps by the JEGO merge module (\ref{['subsec:merge']}). Finally, the coarse-to-fine matching module (C2F) generates the matching results (\ref{['subsec:c2f']}). We show how the JEGO strategy enables Joint, Efficient, Global, and Omnidirectional scanning and merging.
• Figure 4: Sequential scan vs. Joint scan. Sequential scan follows a pattern of internal $\rightarrow$ mutual interaction, similar to self $\rightarrow$ cross attention. Joint scan, by contrast, emphasizes high-frequency mutual interaction, which has proven crucial for feature matching.
• Figure 5: Coarse-to-Fine Matching (C2F) Module.