Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-Time Augmentation for Pose-invariant Face Recognition

Jaemin Jung, Youngjoon Jang, Joon Son Chung

TL;DR

Pose-TTA tackles pose variation in face recognition by performing inference-time pose alignment using a portrait animator to generate pose-matched augmentations, avoiding retraining and frontalisation-induced information loss. It introduces a two-stage framework (head pose alignment via a Face Selector and portrait-augmented synthesis) followed by a weighted embedding aggregation to mitigate distortions from synthetic data. The approach demonstrates consistent improvements across pose-variant benchmarks and multiple architectures without retraining, while keeping frontal performance stable, and uses a simple but effective weighting scheme to balance real and synthetic features. Overall, Pose-TTA offers a training-free, integration-friendly enhancement to existing face recognition pipelines with potential applicability to other face-related tasks in unconstrained environments.

Abstract

The goal of this paper is to enhance face recognition performance by augmenting head poses during the testing phase. Existing methods often rely on training on frontalised images or learning pose-invariant representations, yet both approaches typically require re-training and testing for each dataset, involving a substantial amount of effort. In contrast, this study proposes Pose-TTA, a novel approach that aligns faces at inference time without additional training. To achieve this, we employ a portrait animator that transfers the source image identity into the pose of a driving image. Instead of frontalising a side-profile face -- which can introduce distortion -- Pose-TTA generates matching side-profile images for comparison, thereby reducing identity information loss. Furthermore, we propose a weighted feature aggregation strategy to address any distortions or biases arising from the synthetic data, thus enhancing the reliability of the augmented images. Extensive experiments on diverse datasets and with various pre-trained face recognition models demonstrate that Pose-TTA consistently improves inference performance. Moreover, our method is straightforward to integrate into existing face recognition pipelines, as it requires no retraining or fine-tuning of the underlying recognition models.

Test-Time Augmentation for Pose-invariant Face Recognition

TL;DR

Pose-TTA tackles pose variation in face recognition by performing inference-time pose alignment using a portrait animator to generate pose-matched augmentations, avoiding retraining and frontalisation-induced information loss. It introduces a two-stage framework (head pose alignment via a Face Selector and portrait-augmented synthesis) followed by a weighted embedding aggregation to mitigate distortions from synthetic data. The approach demonstrates consistent improvements across pose-variant benchmarks and multiple architectures without retraining, while keeping frontal performance stable, and uses a simple but effective weighting scheme to balance real and synthetic features. Overall, Pose-TTA offers a training-free, integration-friendly enhancement to existing face recognition pipelines with potential applicability to other face-related tasks in unconstrained environments.

Abstract

The goal of this paper is to enhance face recognition performance by augmenting head poses during the testing phase. Existing methods often rely on training on frontalised images or learning pose-invariant representations, yet both approaches typically require re-training and testing for each dataset, involving a substantial amount of effort. In contrast, this study proposes Pose-TTA, a novel approach that aligns faces at inference time without additional training. To achieve this, we employ a portrait animator that transfers the source image identity into the pose of a driving image. Instead of frontalising a side-profile face -- which can introduce distortion -- Pose-TTA generates matching side-profile images for comparison, thereby reducing identity information loss. Furthermore, we propose a weighted feature aggregation strategy to address any distortions or biases arising from the synthetic data, thus enhancing the reliability of the augmented images. Extensive experiments on diverse datasets and with various pre-trained face recognition models demonstrate that Pose-TTA consistently improves inference performance. Moreover, our method is straightforward to integrate into existing face recognition pipelines, as it requires no retraining or fine-tuning of the underlying recognition models.
Paper Structure (13 sections, 3 equations, 2 figures, 3 tables)

This paper contains 13 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. PROPOSED METHOD
  3. Head Pose Alignment
  4. Test Time Adaptation
  5. EXPERIMENTS
  6. Experimental setup
  7. RESULTS
  8. Effectiveness of Pose-TTA
  9. Comparison with Frontalisation.
  10. Effectiveness of Weighted Feature Aggregation.
  11. Qualitative Comparison
  12. CONCLUSION
  13. ACKNOWLEDGMENTS

Figures (2)

  • Figure 1: An overview of our Pose-TTA framework. In the head pose alignment stage, the source and driving images, $I_s$ and $I_d$, are processed through a portrait animator to generate an augmented face. In the feature aggregation stage, both the original and augmented images are passed through a pre-trained face recongition model to extract and aggregate the face embeddings $\hat{y}$ for verification, yielding the aggregated feature $\hat{y}_{tta}$. Aggr. denotes aggregation.
  • Figure 2: Head pose augmentation quality comparison. The proposed method minimises identity distortions by flipping the source image and applying portrait transformation.