SupScene: Learning Overlap-Aware Global Descriptor for Unconstrained SfM

TL;DR

SupScene tackles the SfM image retrieval bottleneck by learning overlap-aware global descriptors that emphasize geometric overlap over semantic similarity. It introduces an overlap-aware subgraph supervision framework and a DINO-inspired VLAD aggregator (DiVLAD) with a learnable gating mechanism to exploit multi-head attention maps. A Soft SupConLoss is proposed to align descriptor similarities with ground-truth geometric overlap. On GL3D, the approach achieves state-of-the-art retrieval performance and improvements in SfM reconstruction, with competitive parameter efficiency and robust gains from the subgraph-based training and attention-guided aggregation.

Abstract

Image retrieval is a critical step for alleviating the quadratic complexity of image matching in unconstrained Structure-from-Motion (SfM). However, in this context, image retrieval typically focuses more on the image pairs of geometric matchability than on those of semantic similarity, a nuance that most existing deep learning-based methods guided by batched binaries (overlapping vs. non-overlapping pairs) fail to capture. In this paper, we introduce SupScene, a novel solution that learns global descriptors tailored for finding overlapping image pairs of similar geometric nature for SfM. First, to better underline co-visible regions, we employ a subgraph-based training strategy that moves beyond equally important isolated pairs, leveraging ground-truth geometric overlapping relationships with various weights to provide fine-grained supervision via a soft supervised contrastive loss. Second, we introduce DiVLAD, a DINO-inspired VLAD aggregator that leverages the inherent multi-head attention maps from the last block of ViT. And then, a learnable gating mechanism is designed to adaptively utilize these semantically salient cues with visual features, enabling a more discriminative global descriptor. Extensive experiments on the GL3D dataset demonstrate that our method achieves state-of-the-art performance, significantly outperforming NetVLAD while introducing a negligible number of additional trainable parameters. Furthermore, we show that the proposed training strategy brings consistent gains across different aggregation techniques. Code and models are available at https://anonymous.4open.science/r/SupScene-5B73.

Figures (5)

• Figure 1: Pair-wise methods vs. SupScene. Left: conventional pair-wise metric learning relies on isolated triplets (anchor, positive, negative) and optimizes with pairwise losses (e.g., Triplet Loss), capturing only one-to-one relationships. In contrast, our SupScene constructs training batches from overlapping subgraphs of co-visible image pairs in SfM. A weight-shared encoder processes the entire subgraph to produce a dense feature similarity matrix, which is directly supervised by the ground-truth geometric overlap matrix via a soft supervised contrastive loss.
• Figure 2: The overall architecture for extracting our global descriptor. An input image is tokenized into patch tokens by DINOv2; all encoder blocks are frozen except the last (trainable), whose $cls$→patch attention provides per-patch importance. Patch tokens are reshaped into a feature map and aggregated by the proposed DiVLAD, which utilizes multi-head attention maps from DINOv2 to adaptively reweight token contributions during VLAD aggregation, ultimately producing a discriminative compact global descriptor $\mathbf{g}$.
• Figure 3: Detailed architecture of the DiVLAD aggregator. The module extends NetVLAD by introducing a cluster-adaptive gate. This gate analyzes multi-head attention maps to generate a semantic modulation weight, which is fused with the standard spatial soft assignment $(a_{k,n})$. This process guides the final VLAD residual aggregation, ensuring it prioritizes features from semantically salient regions.
• Figure 4: Analysis of training strategy and key hyperparameters. We compare our subgraph-based SupScene framework against the pair-wise.(1)Left: shows the convergence behavior of Recall@25 for DiVLAD aggregator, and(2) details the final accuracy as a function of subgraph and batch size. The star marks the best observed setting.
• Figure 5: Assignment-map visualization. Brighter regions indicate higher assignment scores, representing areas that contribute more significantly to the final global descriptor.