Accurate and Fast Pixel Retrieval with Spatial and Uncertainty Aware Hypergraph Diffusion

TL;DR

The paper tackles the challenge of fast and accurate pixel retrieval in large image databases, where diffusion on scalar graph edges can mispropagate spatial information. It introduces a spatially aware hypergraph diffusion (HD) built on a kNN image graph, with inter-image and intra-image hyperedges to propagate local spatial cues offline and a community-selection mechanism to predict retrieval uncertainty online. HD achieves state-of-the-art image-level and pixel-level performance on ROxford/Paris benchmarks, while offering strong speed and memory efficiency, and its convergence follows the form $( ext{I}- ext{P}')^{-1} ext{Y}^0$. The work also demonstrates practical gains in real-world retrieval by reducing the reliance on expensive spatial verification through uncertainty-driven initialization, making it attractive for scalable search systems.

Abstract

This paper presents a novel method designed to enhance the efficiency and accuracy of both image retrieval and pixel retrieval. Traditional diffusion methods struggle to propagate spatial information effectively in conventional graphs due to their reliance on scalar edge weights. To overcome this limitation, we introduce a hypergraph-based framework, uniquely capable of efficiently propagating spatial information using local features during query time, thereby accurately retrieving and localizing objects within a database. Additionally, we innovatively utilize the structural information of the image graph through a technique we term "community selection". This approach allows for the assessment of the initial search result's uncertainty and facilitates an optimal balance between accuracy and speed. This is particularly crucial in real-world applications where such trade-offs are often necessary. Our experimental results, conducted on the (P)ROxford and (P)RParis datasets, demonstrate the significant superiority of our method over existing diffusion techniques. We achieve state-of-the-art (SOTA) accuracy in both image-level and pixel-level retrieval, while also maintaining impressive processing speed. This dual achievement underscores the effectiveness of our hypergraph-based framework and community selection technique, marking a notable advancement in the field of content-based image retrieval.

Figures (6)

• Figure 1: a) shows a part of an ordinary graph with scalar-weighted, i.e., similarity, edges. Orange frames are the common visible regions among images $\textbf{x}_1$, $\textbf{x}_2$, $\textbf{x}_3$, and $\textbf{x}_4$. Purple frames are the common visible regions between images $\textbf{x}_2$ and $\textbf{x}_5$. $\textbf{x}_3$ and $\textbf{x}_5$ are close neighbors to image $\textbf{x}_2$. While $\textbf{x}_3$ is related to $\textbf{x}_1$ by sharing the orange frame, $\textbf{x}_5$ is not. Utilizing scalar-weighted edges cannot propagate the query in the ordinary graph without this ambiguity issue. b) shows the corresponding hypergraph of a). Inter-image hyperedges $\textbf{e}^1_s$ are shown in yellow, intra-image hyperedges $\textbf{e}^2_k$ are in blue, and local features $\textbf{y}_n$ are in green. A hypergraph path connects local features from $\textbf{y}_1$ to $\textbf{y}_{9}$ in $\textbf{x}_1$ and $\textbf{x}_3$, but no path connects local features in $\textbf{x}_1$ and $\textbf{x}_5$. A large version of this figure is in the appendix.
• Figure 2: Two queries on an image graph with three communities. The numbers on the nodes are their rankings in the initial search. The uncertainty of the initial search result of Q1 is lower than that of Q2 as most retrieved items of Q1 distribute in the same community.
• Figure 3: Illustration of hypergraph diffusion mechanism. The orange boxes with arrows represent the hyperedges, and the blue curved arrows are the ordinary graph edges. In each triplet, the first image and the third image are wrongly connected through the second image. While traditional diffusion and QE methods wrongly propagate the similarity score from the first to the third image through ordinary graph edge, our hypergraph diffusion does not diffuse the similarity scores from the first to the third image by solving the spatial ambiguity problem of propagation.
• Figure 4: Visual examples of how Hypergraph Diffusion (HD) achieves better pixel retrieval results than direct SPatial verification (SP) using the same DELG features. The yellow boxes in query images are the cropped region given by the benchmark. Yellow boxes in SP are the minimum bounding box of the matching points. Yellow boxes with line arrows indicate the diffusion pathes in HD from the query region to the correspondence region in the target database image. In A, an 'easier' database image offers a clear, unoccluded view of the target query landmark, demonstrating improved matching. B highlights how these database images provide viewpoints more aligned with the query image, enhancing object matching. C shows the advantage in scenarios with varying illumination, where 'easier' images assist in achieving more accurate matches. Lastly, D reveals that when the target database image has a more complex background than the query image, direct spatial verification can lead to outlier matchings. In contrast, the 'easier' database images selected by HD provide additional context, thus facilitating more precise pixel retrieval.
• Figure 5: mAP of the query results above the uncertainty threshold. Uncertainty predicts the query quality well.

Theorems & Definitions (2)

• Definition 4.1: Inter-image hyperedge
• Definition 4.2: Intra-image hyperedge