Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval

TL;DR

The paper addresses zero-shot sketch-based image retrieval by bridging sketches and images with auxiliary textual information generated by large-language models. It introduces a three-module framework: (i) Description Generation Module to produce category-specific texts, (ii) Feature Extraction Module using two ViTs and a text encoder to align visual and textual tokens, and (iii) Cross-modal Alignment Module employing cross-attention for local and global alignment, optimized with a global triplet loss $L_{tri}$ and a local relation network loss $L_{rn}$. Inference uses only sketch and image features, while training leverages the textual descriptions to enhance zero-shot generalization. Experiments on three benchmarks show state-of-the-art performance, with ablations and prompt studies confirming the value of textual auxiliary information and appropriate prompt design for GPT-3-generated descriptions.

Abstract

In this paper, we study the problem of zero-shot sketch-based image retrieval (ZS-SBIR). The prior methods tackle the problem in a two-modality setting with only category labels or even no textual information involved. However, the growing prevalence of Large-scale pre-trained Language Models (LLMs), which have demonstrated great knowledge learned from web-scale data, can provide us with an opportunity to conclude collective textual information. Our key innovation lies in the usage of text data as auxiliary information for images, thus leveraging the inherent zero-shot generalization ability that language offers. To this end, we propose an approach called Cross-Modal Attention Alignment Network with Auxiliary Text Description for zero-shot sketch-based image retrieval. The network consists of three components: (i) a Description Generation Module that generates textual descriptions for each training category by prompting an LLM with several interrogative sentences, (ii) a Feature Extraction Module that includes two ViTs for sketch and image data, a transformer for extracting tokens of sentences of each training category, finally (iii) a Cross-modal Alignment Module that exchanges the token features of both text-sketch and text-image using cross-attention mechanism, and align the tokens locally and globally. Extensive experiments on three benchmark datasets show our superior performances over the state-of-the-art ZS-SBIR methods.

Figures (4)

• Figure 1: (i) During training, the sketch, image, and the corresponding text are fed into the model to learn the alignment for the correspondence of a specific region. (ii) For inference, the transferred knowledge is utilized to do the ZS-SBIR.
• Figure 2: Overview. (i) Description Generation Module collects the textual knowledge describing the visual clues for each specific category, which is generated by LLM with some prompts. (ii) Feature Extraction Module takes the sketch, image, and text data and feeds them into the encoders respectively for extracting token-level features by self-attention. (iii) Cross-attention Alignment Module utilizes the cross-attention mechanism to exchange information between sketch-text and image-text tokens. The correspondences of tokens will be measured locally (matching loss) and globally (triplet loss).
• Figure 3: Exemplar comparison retrieval results for the given query sketches and the top 10 retrieved images. Red box denotes false positive, Green box denotes true positive.
• Figure 4: The mAP@all and Prec@100 scores on TU-Berlin with different values of $\lambda_{tri}$ and $\lambda_{rn}$.