Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Identity-Aware Cross-Modal Retrieval: a Dataset and a Baseline

Nicola Messina, Lucia Vadicamo, Leo Maltese, Claudio Gennaro

TL;DR

This paper tackles identity-aware cross-modal retrieval, addressing the difficulty of locating specific individuals within contextual queries by extending vision-language models to long-tail identities. It introduces COCO-PFS, a dataset built by face-swapping COCO images with identities from VGGFace2 and enriching captions with identity names, enabling evaluation of both identity and contextual retrieval. The proposed Id-CLIP framework grounds identities via a face-to-token mapping and employs visual prompt tuning to adapt the image encoder, achieving strong improvements over standard CLIP and CLIP-PAD baselines without per-gallery retraining. The work provides a public dataset and a robust baseline that pave the way for practical identity-aware retrieval in large audiovisual archives, with potential applications in personalization and heritage preservation.

Abstract

Recent advancements in deep learning have significantly enhanced content-based retrieval methods, notably through models like CLIP that map images and texts into a shared embedding space. However, these methods often struggle with domain-specific entities and long-tail concepts absent from their training data, particularly in identifying specific individuals. In this paper, we explore the task of identity-aware cross-modal retrieval, which aims to retrieve images of persons in specific contexts based on natural language queries. This task is critical in various scenarios, such as for searching and browsing personalized video collections or large audio-visual archives maintained by national broadcasters. We introduce a novel dataset, COCO Person FaceSwap (COCO-PFS), derived from the widely used COCO dataset and enriched with deepfake-generated faces from VGGFace2. This dataset addresses the lack of large-scale datasets needed for training and evaluating models for this task. Our experiments assess the performance of different CLIP variations repurposed for this task, including our architecture, Identity-aware CLIP (Id-CLIP), which achieves competitive retrieval performance through targeted fine-tuning. Our contributions lay the groundwork for more robust cross-modal retrieval systems capable of recognizing long-tail identities and contextual nuances. Data and code are available at https://github.com/mesnico/IdCLIP.

Towards Identity-Aware Cross-Modal Retrieval: a Dataset and a Baseline

TL;DR

This paper tackles identity-aware cross-modal retrieval, addressing the difficulty of locating specific individuals within contextual queries by extending vision-language models to long-tail identities. It introduces COCO-PFS, a dataset built by face-swapping COCO images with identities from VGGFace2 and enriching captions with identity names, enabling evaluation of both identity and contextual retrieval. The proposed Id-CLIP framework grounds identities via a face-to-token mapping and employs visual prompt tuning to adapt the image encoder, achieving strong improvements over standard CLIP and CLIP-PAD baselines without per-gallery retraining. The work provides a public dataset and a robust baseline that pave the way for practical identity-aware retrieval in large audiovisual archives, with potential applications in personalization and heritage preservation.

Abstract

Recent advancements in deep learning have significantly enhanced content-based retrieval methods, notably through models like CLIP that map images and texts into a shared embedding space. However, these methods often struggle with domain-specific entities and long-tail concepts absent from their training data, particularly in identifying specific individuals. In this paper, we explore the task of identity-aware cross-modal retrieval, which aims to retrieve images of persons in specific contexts based on natural language queries. This task is critical in various scenarios, such as for searching and browsing personalized video collections or large audio-visual archives maintained by national broadcasters. We introduce a novel dataset, COCO Person FaceSwap (COCO-PFS), derived from the widely used COCO dataset and enriched with deepfake-generated faces from VGGFace2. This dataset addresses the lack of large-scale datasets needed for training and evaluating models for this task. Our experiments assess the performance of different CLIP variations repurposed for this task, including our architecture, Identity-aware CLIP (Id-CLIP), which achieves competitive retrieval performance through targeted fine-tuning. Our contributions lay the groundwork for more robust cross-modal retrieval systems capable of recognizing long-tail identities and contextual nuances. Data and code are available at https://github.com/mesnico/IdCLIP.
Paper Structure (21 sections, 2 equations, 2 figures, 2 tables)

This paper contains 21 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Entity-aware datasets
  4. COCO Person FaceSwap (COCO-PFS) Dataset
  5. Image Pre-Selection
  6. Gender-Ethnicity Detection and Face-swapping
  7. Caption Enrichment
  8. Dataset Overview
  9. Identity-aware Cross-modal Retrieval
  10. Identity-aware CLIP (Id-CLIP)
  11. Overview
  12. Visual Prompt Tuning
  13. Objective
  14. Experiments
  15. Inference Strategies
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Examples from our COCO-PFS dataset showing images obtained by substituting (a) different identities in the same context or (b) the same identity in different contexts. The captions depicted are obtained through two different templates, starting from one of the five original COCO captions.
  • Figure 2: (a). The original CLIP model. In order to digest entity names, should be properly finetuned and specialized to a given gallery $\mathcal{G}$. (b) Id-CLIP solves the problem by factoring out the entity from the caption. Through $\mathcal{G}$, the name is used to lookup the face, which becomes an additional input to the system. The learnable tokens in input to the Visual Encoder implement visual prompt tuning.