Evaluating the Impact of Data Anonymization on Image Retrieval

TL;DR

This work systematically investigates how data anonymization affects Content-Based Image Retrieval (CBIR) using a DINOv2 backbone across CelebA, RVL-CDIP, and the DOKIQ dataset. It proposes an evaluation framework that defines a pseudo ground truth from original data and measures retrieval fidelity after various anonymization methods ($\text{Pixel}$, $\text{Blur}$, $\text{Mask}$) and training adaptions ($\text{A,B,C}$) with $mAP$ and $mnDCG$ as metrics, including $AP$ and DCG-based formulations. Key findings show a persistent retrieval bias toward models trained on non-anonymized data, with performance generally deteriorating as anonymization degree increases, though certain configurations can mitigate loss or even improve downstream tasks when queried with anonymized inputs. The paper offers practical recommendations (e.g., preserve original embeddings/models when possible, case-by-case evaluation) and ethical considerations, highlighting the need for privacy-preserving CBIR approaches that maintain utility under GDPR-like constraints. Overall, mnDCG with anonymized queries emerges as a strong proxy for downstream performance, guiding robust evaluation and design of privacy-compliant CBIR systems.

Abstract

With the growing importance of privacy regulations such as the General Data Protection Regulation, anonymizing visual data is becoming increasingly relevant across institutions. However, anonymization can negatively affect the performance of Computer Vision systems that rely on visual features, such as Content-Based Image Retrieval (CBIR). Despite this, the impact of anonymization on CBIR has not been systematically studied. This work addresses this gap, motivated by the DOKIQ project, an artificial intelligence-based system for document verification actively used by the State Criminal Police Office Baden-Württemberg. We propose a simple evaluation framework: retrieval results after anonymization should match those obtained before anonymization as closely as possible. To this end, we systematically assess the impact of anonymization using two public datasets and the internal DOKIQ dataset. Our experiments span three anonymization methods, four anonymization degrees, and four training strategies, all based on the state of the art backbone Self-Distillation with No Labels (DINO)v2. Our results reveal a pronounced retrieval bias in favor of models trained on original data, which produce the most similar retrievals after anonymization. The findings of this paper offer practical insights for developing privacy-compliant CBIR systems while preserving performance.