CLIP-Branches: Interactive Fine-Tuning for Text-Image Retrieval

TL;DR

The paper tackles imprecision and incompleteness in CLIP-based text-image retrieval caused by pure NN search. It proposes CLIP-Branches, which adds an interactive fine-tuning phase using positive and negative examples and leverages decision-branch models with pre-built indexes for scalable full-catalog classification. Key innovations include a CLIP-based embedding extension with a compact 32-d representation via an 8-bit quantification and KoLeo regularization, and the use of fast decision-branch inference to deliver complete, relevant results with interactive latency. Experiments on datasets up to hundreds of millions of images show improved relevance and completeness while preserving memory efficiency and response times, making the approach practical for real-time multimodal search.

Abstract

The advent of text-image models, most notably CLIP, has significantly transformed the landscape of information retrieval. These models enable the fusion of various modalities, such as text and images. One significant outcome of CLIP is its capability to allow users to search for images using text as a query, as well as vice versa. This is achieved via a joint embedding of images and text data that can, for instance, be used to search for similar items. Despite efficient query processing techniques such as approximate nearest neighbor search, the results may lack precision and completeness. We introduce CLIP-Branches, a novel text-image search engine built upon the CLIP architecture. Our approach enhances traditional text-image search engines by incorporating an interactive fine-tuning phase, which allows the user to further concretize the search query by iteratively defining positive and negative examples. Our framework involves training a classification model given the additional user feedback and essentially outputs all positively classified instances of the entire data catalog. By building upon recent techniques, this inference phase, however, is not implemented by scanning the entire data catalog, but by employing efficient index structures pre-built for the data. Our results show that the fine-tuned results can improve the initial search outputs in terms of relevance and accuracy while maintaining swift response times

Figures (4)

• Figure 1: Search process of CLIP-Branches. Traditional text-to-image search engines only consist of the steps ❶ to ❸ while CLIP-Branches adds a fine-tuning stage to refine the initial search results (Steps ❹-❼). Index structures are employed during the search for faster execution.
• Figure 2: Feature extraction for CLIP-Branches. The initial text and image data are transformed into 512-dimensional embeddings using CLIP. These embeddings are refined via a custom head module, followed by 8-bit quantization, resulting in a 32-dimensional 8-bit embedding. The reduction in data size is exemplified below using the LAION data set.
• Figure 3: Number of positive training instances required for Decision Branches models to surpass NN search results among different image classification benchmark data sets.
• Figure 4: Demonstration of CLIP-Branches search workflow. The user initiates a search with a query string and receives top $k$-initial results. These results are then labeled as positive (green) or negative (red) based on user preference, guiding the fine-tuning of the search. The fine-tuned results reflect more completeness and higher accuracy.