Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nearest Neighbor Normalization Improves Multimodal Retrieval

Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann, Tristan Thrush

TL;DR

A simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called NNN, which shows an improvement on retrieval metrics in both text retrieval and image retrieval.

Abstract

Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.

Nearest Neighbor Normalization Improves Multimodal Retrieval

TL;DR

A simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called NNN, which shows an improvement on retrieval metrics in both text retrieval and image retrieval.

Abstract

Multimodal models leverage large-scale pre-training to achieve strong but still imperfect performance on tasks such as image captioning, visual question answering, and cross-modal retrieval. In this paper, we present a simple and efficient method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization (NNN). We show an improvement on retrieval metrics in both text retrieval and image retrieval for all of the contrastive models that we tested (CLIP, BLIP, ALBEF, SigLIP, BEiT) and for both of the datasets that we used (MS-COCO and Flickr30k). NNN requires a reference database, but does not require any training on this database, and can even increase the retrieval accuracy of a model after finetuning.

Paper Structure

This paper contains 25 sections, 10 equations, 5 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Nearest Neighbor Normalization
  3. Experiments
  4. Retrieval performance
  5. Accuracy.
  6. Efficiency.
  7. Correcting image and caption bias
  8. Reducing gender bias in image retrieval
  9. Conclusion
  10. Limitations
  11. Ethical considerations
  12. Acknowledgements
  13. Baselines
  14. DBNorm
  15. QBNorm
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Method overview.applies an additive correction at inference time, using bias scores estimated from a reference database of queries.
  • Figure 2: Distribution of COCO captions matched to each image during image retrieval. A base CLIP model contains many hubs that match over 100 captions, while the distribution after shows fewer hubs, on par with finetuning on COCO.
  • Figure 3: decreases gender bias in image retrieval. (L) Top 10 retrieved Visogender images for an example query, before (top) and after (bottom) debiasing. (R) Distribution of image retrieval bias across occupations.
  • Figure A1: Distribution of COCO captions matched to each image during image retrieval for BLIP crossmodal Applying to the cross-attention model does not significantly affect the distribution: a Kolmogorov-Smirnov test has a p-value of 0.846. (One caption was chosen per image due to compute constraints.)
  • Figure A2: Distribution of captions matched per image for image retrieval (left), and images matched per caption for text retrieval (right).