When Machine Learning Models Leak: An Exploration of Synthetic Training Data

TL;DR

The paper addresses privacy risks when releasing a propensity-to-move classifier and evaluates whether training on synthetic data mitigates model-inversion attribute inference attacks. It formalizes a label-only MIA with marginals (LOMIA+Marginals) and compares against a marginals-only baseline, using a real-world dataset and CART-based synthetic data generation. The experiments show that models trained on synthetic data retain similar predictive performance to original-data models and can reduce leakage from attribute inference attacks, although marginals still contribute substantially to attacks. The findings suggest synthetic-data training can help protect against attribute disclosure in released models, while highlighting the need for broader threat models and privacy metrics in future work.

Abstract

We investigate an attack on a machine learning model that predicts whether a person or household will relocate in the next two years, i.e., a propensity-to-move classifier. The attack assumes that the attacker can query the model to obtain predictions and that the marginal distribution of the data on which the model was trained is publicly available. The attack also assumes that the attacker has obtained the values of non-sensitive attributes for a certain number of target individuals. The objective of the attack is to infer the values of sensitive attributes for these target individuals. We explore how replacing the original data with synthetic data when training the model impacts how successfully the attacker can infer sensitive attributes.