How Worrying Are Privacy Attacks Against Machine Learning?

TL;DR

The paper investigates privacy risks associated with releasing trained ML models, focusing on membership inference, property inference, and reconstruction attacks. It argues that real-world conditions—non-exhaustive data, attribute diversity, and access constraints—curtail the effectiveness of these attacks, especially for well-generalized models. It discusses gradient-inversion attacks in federated settings and notes that many reconstruction methods struggle to decide true membership without original data, implying high costs and limited practicality. The result is a more optimistic view of trustworthy ML and a call to avoid overbearing privacy defenses that unduly degrade utility, particularly in the EU regulatory context.

Abstract

In several jurisdictions, the regulatory framework on the release and sharing of personal data is being extended to machine learning (ML). The implicit assumption is that disclosing a trained ML model entails a privacy risk for any personal data used in training comparable to directly releasing those data. However, given a trained model, it is necessary to mount a privacy attack to make inferences on the training data. In this concept paper, we examine the main families of privacy attacks against predictive and generative ML, including membership inference attacks (MIAs), property inference attacks, and reconstruction attacks. Our discussion shows that most of these attacks seem less effective in the real world than what a prima face interpretation of the related literature could suggest.