SimpleMatch: A Simple and Strong Baseline for Semantic Correspondence

TL;DR

SimpleMatch addresses the efficiency bottleneck in semantic correspondence by tackling irreversible keypoint feature fusion caused by deep downsampling. It introduces a lightweight upsampling decoder and a multi-scale loss, complemented by memory-saving sparse matching and window-based localization to operate effectively at low resolutions. On SPair-71k, it achieves a high $PCK@0.1$ of $84.1\%$ at $252\times252$, while using a $3.3\times$ smaller input and running at 65 images per second with modest memory, demonstrating strong practical efficiency alongside accuracy. Overall, SimpleMatch provides a simple, robust baseline that balances performance and resource usage, making high-quality semantic correspondence more accessible for real-world applications.

Abstract

Recent advances in semantic correspondence have been largely driven by the use of pre-trained large-scale models. However, a limitation of these approaches is their dependence on high-resolution input images to achieve optimal performance, which results in considerable computational overhead. In this work, we address a fundamental limitation in current methods: the irreversible fusion of adjacent keypoint features caused by deep downsampling operations. This issue is triggered when semantically distinct keypoints fall within the same downsampled receptive field (e.g., 16x16 patches). To address this issue, we present SimpleMatch, a simple yet effective framework for semantic correspondence that delivers strong performance even at low resolutions. We propose a lightweight upsample decoder that progressively recovers spatial detail by upsampling deep features to 1/4 resolution, and a multi-scale supervised loss that ensures the upsampled features retain discriminative features across different spatial scales. In addition, we introduce sparse matching and window-based localization to optimize training memory usage and reduce it by 51%. At a resolution of 252x252 (3.3x smaller than current SOTA methods), SimpleMatch achieves superior performance with 84.1% PCK@0.1 on the SPair-71k benchmark. We believe this framework provides a practical and efficient baseline for future research in semantic correspondence. Code is available at: https://github.com/hailong23-jin/SimpleMatch.

Figures (6)

• Figure 1: Feature map visualizations at different scales. The red dots represent keypoints.
• Figure 2: Illustration of SimpleMatch structure. The architecture consists solely of a feature extractor and a lightweight upsampling decoder. After obtaining the source and target feature maps, we perform sparse matching and employ window-based localization to enhance training efficiency.
• Figure 3: Illustration of the lightweight decoder.
• Figure 4: The PCK@$\alpha$ curves of SimpleMatch and previous methods on SPair-71k dataset.
• Figure 5: Visualization of semantic correspondence using different feature map resolutions. From left to right, we show the semantic correspondence performance at 16x16, 32x32 and 64x64 resolutions. Green lines indicate correct matches, while red lines denote incorrect matches. The results are evaluated on PCK@0.1.