Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPOT-Face: Forensic Face Identification using Attention Guided Optimal Transport

Ravi Shankar Prasad, Dinesh Singh

TL;DR

This paper tackles cross-domain forensic face identification when only skull or sketch evidence is available, introducing SPOT-Face, a graph-based framework that converts images into superpixel graphs and aligns skull/sketch graphs with face graphs via cross-attention and optimal transport. By training with a triplet loss across skull-face and sketch-face pairs, and leveraging multiple GNN backbones (notably GraphTransformer), the method learns a unified embedding space that bridges modality gaps. Empirical results on IIT_Mandi_S2F and CUFS show significant gains in recall and mAP, with strong retrieval performance and favorable qualitative analyses, indicating practical utility for forensic identification. Overall, SPOT-Face offers a principled, unified cross-domain representation for skull/sketch-to-face retrieval, advancing automated forensic matching where traditional biometrics are unavailable.

Abstract

Person identification in forensic investigations becomes very challenging when common identification means for DNA (i.e., hair strands, soft tissue) are not available. Current methods utilize deep learning methods for face recognition. However, these methods lack effective mechanisms to model cross-domain structural correspondence between two different forensic modalities. In this paper, we introduce a SPOT-Face, a superpixel graph-based framework designed for cross-domain forensic face identification of victims using their skeleton and sketch images. Our unified framework involves constructing a superpixel-based graph from an image and then using different graph neural networks(GNNs) backbones to extract the embeddings of these graphs, while cross-domain correspondence is established through attention-guided optimal transport mechanism. We have evaluated our proposed framework on two publicly available dataset: IIT\_Mandi\_S2F (S2F) and CUFS. Extensive experiments were conducted to evaluate our proposed framework. The experimental results show significant improvement in identification metrics ( i.e., Recall, mAP) over existing graph-based baselines. Furthermore, our framework demonstrates to be highly effective for matching skulls and sketches to faces in forensic investigations.

SPOT-Face: Forensic Face Identification using Attention Guided Optimal Transport

TL;DR

This paper tackles cross-domain forensic face identification when only skull or sketch evidence is available, introducing SPOT-Face, a graph-based framework that converts images into superpixel graphs and aligns skull/sketch graphs with face graphs via cross-attention and optimal transport. By training with a triplet loss across skull-face and sketch-face pairs, and leveraging multiple GNN backbones (notably GraphTransformer), the method learns a unified embedding space that bridges modality gaps. Empirical results on IIT_Mandi_S2F and CUFS show significant gains in recall and mAP, with strong retrieval performance and favorable qualitative analyses, indicating practical utility for forensic identification. Overall, SPOT-Face offers a principled, unified cross-domain representation for skull/sketch-to-face retrieval, advancing automated forensic matching where traditional biometrics are unavailable.

Abstract

Person identification in forensic investigations becomes very challenging when common identification means for DNA (i.e., hair strands, soft tissue) are not available. Current methods utilize deep learning methods for face recognition. However, these methods lack effective mechanisms to model cross-domain structural correspondence between two different forensic modalities. In this paper, we introduce a SPOT-Face, a superpixel graph-based framework designed for cross-domain forensic face identification of victims using their skeleton and sketch images. Our unified framework involves constructing a superpixel-based graph from an image and then using different graph neural networks(GNNs) backbones to extract the embeddings of these graphs, while cross-domain correspondence is established through attention-guided optimal transport mechanism. We have evaluated our proposed framework on two publicly available dataset: IIT\_Mandi\_S2F (S2F) and CUFS. Extensive experiments were conducted to evaluate our proposed framework. The experimental results show significant improvement in identification metrics ( i.e., Recall, mAP) over existing graph-based baselines. Furthermore, our framework demonstrates to be highly effective for matching skulls and sketches to faces in forensic investigations.
Paper Structure (18 sections, 7 equations, 5 figures, 3 tables)

This paper contains 18 sections, 7 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Skull-to-face
  4. Sketch-to-face
  5. Dataset
  6. IIT_Mandi_S2F (S2F).
  7. CUFS.
  8. Methodology
  9. Graph Structure Data for Images
  10. Learning Cross-Domain Identity Representation
  11. Graph-Based Cross-Modal Alignment
  12. Image retrieval framework
  13. Experiments
  14. Results
  15. Quantitative Results:
  16. ...and 3 more sections

Figures (5)

  • Figure 1: (a) shows paired face and their X-ray samples from IIT_Mandi_S2F dataset where (b) represents the paired face and their sketch samples from CUFS dataset with superpixel representation and their graph.
  • Figure 2: Visualization of embeddings of S2F and CUFS dataset. This 2D t-SNE shows that S2F (skull-to-face) dataset have larger domain gap than CUFS (sketch-to-face) dataset. (Best viewed in colors)
  • Figure 3: In this framework, different GNNs serves as the backbone to extract features from the input skull-face and sketch-face graphs. First, skull, sketch and their respective face images are converted to superpixels using the SLIC method, and from superpixel segmentation of images, we get the graphs by treating each segment as a node. Then, this graph is input to the trainable GNN module. Finally, to enhance the features of the skull, sketch and face graphs, we leverage a cross-attention and optimal transport module. We perform feature matching by minimizing the triplet loss. (Best viewed in colors)
  • Figure 4: (a) and (b) show ROC_AUC curve for IIT_Mandi_S2F and CUFS dataset with different modules, respectively. (Best viewed in colors)
  • Figure 5: Qualitative evaluation for face retrieval showing top-5 retrieved faces. Where the green border face is the true face for the given query skull and sketch in IIT_Mandi_S2F and CUFS dataset. (Best viewed in colors)