Footprints of Data in a Classifier: Understanding the Privacy Risks and Solution Strategies

TL;DR

This paper addresses privacy risks from training-data footprints embedded in classifiers, tying the issue to GDPR Article 17 and the Right to Erasure. It develops a theoretical vulnerability framework showing universal susceptibility under class imbalance and distribution shifts, and backs it with empirical analysis across diverse datasets and classifiers. The study investigates data obfuscation as a mitigation and introduces a privacy-performance trade-off metric to guide practical deployment decisions. Overall, the work provides actionable insights for classifier selection, data curation, and privacy-preserving strategies in real-world AI systems.

Abstract

The widespread deployment of Artificial Intelligence (AI) across government and private industries brings both advancements and heightened privacy and security concerns. Article 17 of the General Data Protection Regulation (GDPR) mandates the Right to Erasure, requiring data to be permanently removed from a system to prevent potential compromise. While existing research primarily focuses on erasing sensitive data attributes, several passive data compromise mechanisms remain underexplored and unaddressed. One such issue arises from the residual footprints of training data embedded within predictive models. Performance disparities between test and training data can inadvertently reveal which data points were part of the training set, posing a privacy risk. This study examines how two fundamental aspects of classifier systems - training data quality and classifier training methodology - contribute to privacy vulnerabilities. Our theoretical analysis demonstrates that classifiers exhibit universal vulnerability under conditions of data imbalance and distributional shifts. Empirical findings reinforce our theoretical results, highlighting the significant role of training data quality in classifier susceptibility. Additionally, our study reveals that a classifier's operational mechanism and architectural design impact its vulnerability. We further investigate mitigation strategies through data obfuscation techniques and analyze their impact on both privacy and classification performance. To aid practitioners, we introduce a privacy-performance trade-off index, providing a structured approach to balancing privacy protection with model effectiveness. The findings offer valuable insights for selecting classifiers and curating training data in diverse real-world applications.

Figures (7)

• Figure 1: Intuition behind vulnerability estimation. Training data is used in building a classifier. When the predictions obtained for the training data and the test data show similar veracity, the classifier is said to be not vulnerable. If the veracity of prediction is substantially higher for the training data, the classifier is identified as vulnerable.
• Figure 2: Vulnerability of classifiers on different datasets. The figure has a central component that shows the vulnerabilities, and the side panel shows the color legend. Each row of the main figure corresponds to a dataset, and each column corresponds to a classifier. The vulnerabilities of the imbalanced datasets are arranged in the top three rows (Churn, Contraceptive, and Customer), and the bottom two rows show the vulnerabilities of the balanced datasets. The vulnerability values of the vulnerable classifiers (Decision Tree, Random Forest, XGBoost, KNN, and MLP-deep) are reported on the left side of the table, and the non-vulnerable classifiers (SGD, AdaBoost, GNB, Logistic Regression, and MLP-shallow) are reported on the right side of the table.
• Figure 3: Analysis of Churn dataset showing vulnerability as a function of (a) number of layers and (b) number of neurons. This dataset has a substantial amount of imbalance between its two classes. Bias in the data couples with the increasing number of neurons and the increasing number of layers, and contributes to a positive correlation of either with vulnerability.
• Figure 4: Analysis of Contraceptive dataset showing vulnerability as a function of (a) number of layers and (b) number of neurons. This dataset has an admissible degree of imbalance between its classes. Bias in the data, coupled with an increasing number of neurons and an increasing number of layers, contributes to an increase in vulnerability.
• Figure 5: Analysis of Customer dataset showing vulnerability as a function of (a) number of layers and (b) number of neurons. This dataset has a substantial amount of imbalance between its three classes. Bias in the data couples with the increasing number of neurons and the increasing number of layers, and renders a positive correlation of either with vulnerability.