LOCORE: Image Re-ranking with Long-Context Sequence Modeling

TL;DR

LOCORE reframes image re-ranking as a long-context, list-wise problem over local descriptors, enabling joint reasoning across the query and an entire shortlist. It leverages a Longformer-based backbone with a sliding-window strategy to process large candidate sets, producing per-image scores via token-level predictions and aggregation. Through gallery-shuffled training, global query attention, and effective inference with sliding windows, LOCORE achieves state-of-the-art re-ranking on ROxf, RPar, SOP, In-Shop, and CUB while maintaining competitive latency and memory. This approach widens the practical impact of local-descriptor re-ranking by capturing cross-image interactions and transitive relationships, offering robust gains across diverse benchmarks.

Abstract

We introduce LOCORE, Long-Context Re-ranker, a model that takes as input local descriptors corresponding to an image query and a list of gallery images and outputs similarity scores between the query and each gallery image. This model is used for image retrieval, where typically a first ranking is performed with an efficient similarity measure, and then a shortlist of top-ranked images is re-ranked based on a more fine-grained similarity measure. Compared to existing methods that perform pair-wise similarity estimation with local descriptors or list-wise re-ranking with global descriptors, LOCORE is the first method to perform list-wise re-ranking with local descriptors. To achieve this, we leverage efficient long-context sequence models to effectively capture the dependencies between query and gallery images at the local-descriptor level. During testing, we process long shortlists with a sliding window strategy that is tailored to overcome the context size limitations of sequence models. Our approach achieves superior performance compared with other re-rankers on established image retrieval benchmarks of landmarks (ROxf and RPar), products (SOP), fashion items (In-Shop), and bird species (CUB-200) while having comparable latency to the pair-wise local descriptor re-rankers.

Figures (7)

• Figure 1: Overview of pair-wise vs our proposed list-wise re-ranking. Red circles denote the locations of input local descriptors. Left: The pair-wise re-ranker gets a high score for a positive image since it clearly depicts the same structure at the top of the tower as the query, while a different positive image gets a low score because the top of the tower is not as clearly visible. Right: Our long-context re-ranker can output a high score for both positive image results since it can exploit the transitive relationship between these images as the two gallery images also share common local descriptors.
• Figure 2: Overview of training and inference under LoCoRe when re-ranking three candidate gallery images. In practice, we re-rank in one inference step up to 100 gallery images. At training time, the model is trained to optimize a binary cross-entropy loss on each gallery image token. At inference time, the token scores of each gallery image get aggregated to facilitate a re-ranked gallery image list.
• Figure 3: Attention pattern of LoCoRe when the number of descriptors for each image is $L=4$ and the local window size is $W=1$. In our experiments we actually use a larger number of local feature descriptors and window size.
• Figure 4: Illustration of sliding window re-ranking for $N=8$ gallery images with a list-wise re-ranker that can re-rank $K=4$ images each forward pass. Blue blocks represent the re-ranking window of the current forward pass, and it slides to the next window with a stride of $S=2$ images. Brown blocks indicate the re-ranking for the current forward pass is completed.
• Figure A: Average precision per query scatter plot on $\mathcal{R}$Oxf+1M Hard for global-only vs. AMES (Left), global-only vs. LoCoRe-small (Middle) and AMES vs. LoCoRe-small (Right). Global descriptors are from RN101-Superglobal, which by itself achieves mAP=61.4%. Re-ranking is performed for top-100 candidates, and the color bar indicates the number of positive images in the shortlist for each query.