A False Sense of Privacy: Towards a Reliable Evaluation Methodology for the Anonymization of Biometric Data

TL;DR

Biometric anonymization is essential for protecting privacy but relies on flawed evaluation methodologies that assume weak adversaries and large identity sets. This work introduces a pessimistic, worst-case evaluation framework by (i) training recognition systems on anonymized data (parrot learning), (ii) testing across multiple recognition architectures, and (iii) constructing harder evaluation datasets via identity reduction and strategic selection. Through extensive experiments on face and gait data, the authors show improved reliability of anonymization assessments and demonstrate that no single recognizer reliably dominates across all anonymizations. The findings offer practical guidelines to prevent false guarantees of privacy and to foster more robust evaluation standards in biometric privacy research.

Abstract

Biometric data contains distinctive human traits such as facial features or gait patterns. The use of biometric data permits an individuation so exact that the data is utilized effectively in identification and authentication systems. But for this same reason, privacy protections become indispensably necessary. Privacy protection is extensively afforded by the technique of anonymization. Anonymization techniques protect sensitive personal data from biometrics by obfuscating or removing information that allows linking records to the generating individuals, to achieve high levels of anonymity. However, our understanding and possibility to develop effective anonymization relies, in equal parts, on the effectiveness of the methods employed to evaluate anonymization performance. In this paper, we assess the state-of-the-art methods used to evaluate the performance of anonymization techniques for facial images and for gait patterns. We demonstrate that the state-of-the-art evaluation methods have serious and frequent shortcomings. In particular, we find that the underlying assumptions of the state-of-the-art are quite unwarranted. State-of-the-art methods generally assume a difficult recognition scenario and thus a weak adversary. However, that assumption causes state-of-the-art evaluations to grossly overestimate the performance of the anonymization. Therefore, we propose a strong adversary which is aware of the anonymization in place. We improve the selection process for the evaluation dataset, and we reduce the numbers of identities contained in the dataset while ensuring that these identities remain easily distinguishable from one another. Our novel evaluation methodology surpasses the state-of-the-art because we measure worst-case performance and so deliver a highly reliable evaluation of biometric anonymization techniques.

Figures (19)

• Figure 1: Dataset use through the phases of our recognition systems for face and gait recognition.
• Figure 2: Simplified example for Genuine and Imposter scores of an identity A in a 2D projection of the feature space.
• Figure 3: Sample pose of motion-captured gait information, represented as point-light walker.
• Figure 4: Schematic overview of the evaluation framework architecture, excluding pre-training for simplicity
• Figure 5: Example image for each of the face anonymization techniques we assess.