If At First You Don't Succeed: Test Time Re-ranking for Zero-shot, Cross-domain Retrieval

TL;DR

The paper tackles zero-shot, cross-domain image retrieval by introducing Iterative Cluster-free Re-ranking (ICFRR), a test-time method that leverages gallery-gallery relationships to reveal semantic connections not captured by direct visual similarity. Combined with a ViT-based backbone (ViT-x) and cross-domain loss functions, ICFRR updates query-gallery scores iteratively via $s^{(t+1)}(x_Q^A,x_i^B) = s^{(t)}(x_Q^A,x_i^B) + \beta \Delta_i^{(t)}$, where $\Delta_i^{(t)}$ aggregates top-$K_q$ gallery relationships using $\alpha[r] = 1 - (r-1)/(G-1)$ for top-$K_g$ ranks. The approach achieves state-of-the-art results on Sketchy, TU-Berlin, and QuickDraw SBIR benchmarks and demonstrates strong transfer to non-SBIR cross-domain tasks such as Office-Home, all while remaining backbone-agnostic and scalable. This work provides a practical, plug-and-play re-ranking mechanism that robustly bridges semantic gaps across domains and holds promise for broad cross-domain retrieval applications. The method delivers practical impact by improving retrieval accuracy without requiring task-specific engineering, making it suitable for diverse cross-domain scenarios.

Abstract

In this paper, we introduce a novel method for zero-shot, cross-domain image retrieval. Our key contribution is a test-time Iterative Cluster-free Re-ranking process that leverages gallery-gallery feature information to establish semantic links between query and gallery images. This enables the retrieval of relevant images even when they do not exhibit similar visual features but share underlying semantic concepts. This can be combined with any pre-existing cross-domain feature extraction backbone to improve retrieval performance. However, when combined with a carefully chosen Vision Transformer backbone and combination of zero-shot retrieval losses, our approach yields state-of-the-art results on the Sketchy, TU-Berlin and QuickDraw sketch-based retrieval benchmarks. We show that our re-ranking also improves performance with other backbones and outperforms other re-ranking methods applied with our backbone. Importantly, unlike many previous methods, none of the components in our approach are engineered specifically towards the sketch-based image retrieval task - it can be generally applied to any cross-domain, zero-shot retrieval task. We therefore also present new results on zero-shot cartoon-to-photo and art-to-product retrieval using the Office-Home dataset. Project page: finlay-hudson.github.io/icfrr, code available at: github.com/finlay-hudson/ICFRR

Figures (15)

• Figure 1: Re-ranking rationale: (A) is a sketch of an ornate wooden door while (B) is a modern glass sliding door. These images share almost no visual similarities, only semantic similarities. Re-ranking at test-time can help bridge the visual gap by exploiting relationships between same-domain gallery images (C, D).
• Figure 2: Overview of the architecture and operation of our approach. During training, image pairs are fed into a Vision Transformer model, producing embeddings and class predictions. These embeddings are used for both the Cross-domain Hard Example Triplet loss and the Cross Entropy loss. At test time, individual images are passed through a Vision Transformer to generate embeddings. Gallery image embeddings are then ranked based on their distances from a Query image embeddings, followed by re-ranking using our Iterative Cluster-free Re-ranking process.
• Figure 3: Overview of proposed iterative re-ranking approach. Initial query-gallery ranks (a) incorporate information from gallery-gallery ranks (b). Gallery images ranked highly against gallery images with high query-gallery rank (c) are rewarded and move up the ranked list when scores are updated (d).
• Figure 4: Results of re-ranking. For each query sketch the top row relates to the original output while the row below is the re-ranked output. Correct matches are shown in green, incorrect in red.
• Figure 5: Varying $K_q$, $K_g$ and $\beta$ on Sketchy dataset yelamarthi2018zero; a shows full dataset, b a single query class with 10 gallery examples.