Combined CNN and ViT features off-the-shelf: Another astounding baseline for recognition

TL;DR

This work builds on the previous study using off-the-shelf Convolutional Neural Network and extends it to include the more recently proposed Vision Transformers, demonstrating that middle-layer features from CNNs and ViTs are a suitable way to recognize individuals based on periocular images.

Abstract

We apply pre-trained architectures, originally developed for the ImageNet Large Scale Visual Recognition Challenge, for periocular recognition. These architectures have demonstrated significant success in various computer vision tasks beyond the ones for which they were designed. This work builds on our previous study using off-the-shelf Convolutional Neural Network (CNN) and extends it to include the more recently proposed Vision Transformers (ViT). Despite being trained for generic object classification, middle-layer features from CNNs and ViTs are a suitable way to recognize individuals based on periocular images. We also demonstrate that CNNs and ViTs are highly complementary since their combination results in boosted accuracy. In addition, we show that a small portion of these pre-trained models can achieve good accuracy, resulting in thinner models with fewer parameters, suitable for resource-limited environments such as mobiles. This efficiency improves if traditional handcrafted features are added as well.

Figures (6)

• Figure 1: Images from one user of the UBIPr database (relative scale difference between images is shown). Note that images of the right eye are flipped horizontally to match the same orientation as the left image (Section \ref{['sect:db_protocol']}).
• Figure 2: Extraction of periocular features from different CNN/ViT layers.
• Figure 3: Verification results (EER) of different CNN and ViT layers with the various vector normalization techniques employed.
• Figure 4: Learnables of the CNNs and ViTs up to a certain layer.
• Figure 5: Fusion results of CNN with ViT layers using L2 normalization per channel (optimal case of Figure \ref{['fig:resuls-individual']}). Grey values of the images are scaled between 0% (black) and 25% EER (white).