SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval

TL;DR

SQUARE introduces a training-free, two-stage framework for zero-shot composed image retrieval that leverages Multimodal Large Language Models to augment the query (SQAF) and to jointly reason over top candidates in a grid (EBR). SQAF blends CLIP-based embeddings with an MLLM-generated target caption to form a semantically richer global query, while EBR performs efficient, single-pass reranking over a grid of candidates to refine fine-grained alignment. Across CIRR, CIRCO, FashionIQ, and GeneCIS, SQUARE achieves state-of-the-art or competitive results, particularly with smaller backbones, and benefits substantially from the MLLM-based reranking. The approach demonstrates practical, scalable training-free CIR with clear gains in both global ranking and local, semantically coherent reranking, highlighting the value of integrating MLLMs for query understanding and candidate comparison in zero-shot settings.

Abstract

Composed Image Retrieval (CIR) aims to retrieve target images that preserve the visual content of a reference image while incorporating user-specified textual modifications. Training-free zero-shot CIR (ZS-CIR) approaches, which require no task-specific training or labeled data, are highly desirable, yet accurately capturing user intent remains challenging. In this paper, we present SQUARE, a novel two-stage training-free framework that leverages Multimodal Large Language Models (MLLMs) to enhance ZS-CIR. In the Semantic Query-Augmented Fusion (SQAF) stage, we enrich the query embedding derived from a vision-language model (VLM) such as CLIP with MLLM-generated captions of the target image. These captions provide high-level semantic guidance, enabling the query to better capture the user's intent and improve global retrieval quality. In the Efficient Batch Reranking (EBR) stage, top-ranked candidates are presented as an image grid with visual marks to the MLLM, which performs joint visual-semantic reasoning across all candidates. Our reranking strategy operates in a single pass and yields more accurate rankings. Experiments show that SQUARE, with its simplicity and effectiveness, delivers strong performance on four standard CIR benchmarks. Notably, it maintains high performance even with lightweight pre-trained, demonstrating its potential applicability.

Figures (9)

• Figure 1: Overview of SQUARE, our training-free framework for zero-shot composed image retrieval (ZS-CIR). SQUARE takes a coarse-to-fine strategy. In stage 1, Semantic Query-Augmented Fusion (SQAF) enriches the embedding-based composed query, formed by fusing the embeddings of the reference image and modification text from a VLM, by incorporating an MLLM-generated target image caption. This addition improves retrieval accuracy while also making the retrieval process interpretable. In stage 2, Efficient Batch Reranking (EBR) leverages the MLLM’s multimodal reasoning to refine the ranking. The top-$K$ candidate images from SQAF's output are arranged in a grid, with each image annotated with a distinct label and a bounding box. This presentation allows the MLLM to assess all candidates jointly in a single forward pass, yielding faster inference and more precise reranking.
• Figure 2: Example prompt used for generating the imagined target image caption. Our prompt is composed of three handcrafted few-shot examples and a set of explicit rules that guide the MLLM to generate a concise and concrete description of the target image based on a given reference image and textual modification.
• Figure 3: An example grid image used in the reranking process. Each candidate image is marked with a colored bounding box and a numeric label at the top-left corner. These identifiers enable the MLLM to reference specific images during reasoning and generate an updated ranking based on the image relevance to the user's intent.
• Figure 4: Example prompt for the reranking stage. This prompt provides the MLLM with the reference image, the textual modification, and a grid of candidate images. The MLLM's task is to reason over the visual differences and produce an updated ranking based on the candidates' alignment with the user's intent.
• Figure 5: Example prompt for reranking with image captions generated in SQAF. The prompt guides the MLLM to reason over visual differences and update the ranking based on how well the candidates align with the user’s intent expressed as the captions.