Correlation inference attacks against machine learning models

TL;DR

This work reveals that trained ML models can leak dataset-level correlations among input variables, not just individual records. It introduces two correlation inference attacks: a model-less baseline leveraging correlation constraints and a model-based attack that exploits black-box model outputs by training a meta-classifier on synthetic data generated with Gaussian copulas. The authors demonstrate that LR and MLP models leak correlations on synthetic and real-world tabular data, with the model-based attack achieving high accuracy and often surpassing DP defenses or query-limiting mitigations. They further show that extracted correlations can power attribute inference attacks (CI-AIA), underscoring broader privacy risks. Overall, the study provides a principled framework for quantifying and exploiting dataset-level leakage and calls for auditing and stronger defenses against such correlations in practice.

Abstract

Despite machine learning models being widely used today, the relationship between a model and its training dataset is not well understood. We explore correlation inference attacks, whether and when a model leaks information about the correlations between the input variables of its training dataset. We first propose a model-less attack, where an adversary exploits the spherical parametrization of correlation matrices alone to make an informed guess. Second, we propose a model-based attack, where an adversary exploits black-box model access to infer the correlations using minimal and realistic assumptions. Third, we evaluate our attacks against logistic regression and multilayer perceptron models on three tabular datasets and show the models to leak correlations. We finally show how extracted correlations can be used as building blocks for attribute inference attacks and enable weaker adversaries. Our results raise fundamental questions on what a model does and should remember from its training set.

Figures (16)

• Figure 1: Illustration of correlation constraints $(\rho(X_1, Y), \rho(X_2, Y))$ that lead to only one possible bin for the unknown correlation $\rho(X_1, X_2)$. Here, we show results for $N_B=3$ possible bins.
• Figure 2: Accuracy of the correlation inference attack on $n=3$ variables as a function of the correlation constraints $\rho(X_1, Y)$ and $\rho(X_2, Y)$. We show results of (A) the model-less attack and of the model-based against two different target models: (B) logistic regression and (C) multilayer perceptron. The color of each cell indicates the accuracy of our correlation inference attack applied to models trained on synthetic datasets whose correlations $\rho(X_1, Y)$ and $\rho(X_2, Y)$ belong to the region defined by the cell. There are $200\times200$ cells in total.
• Figure 3: Attack accuracy for different scenarios and number of variables in the dataset $n$. We consider three different attack scenarios: (S1) the adversary knows the correlations between the target variables and the output variable $\rho(X_1,Y)$ and $\rho(X_2,Y)$, (S2) the adversary knows the correlations between all the input variables and the output variable $\rho(X_i,Y)_{i=1,\ldots,n-1}$, and (S3) the adversary knows all the correlations between the variables except for the target correlation $\rho(X_1,X_2)$. We compute the accuracy (with 95% confidence interval) over 1000 target models.
• Figure 4: Impact of mitigations on the accuracy of our attack against Logistic regression models. We report results for two mitigations: limiting the number of black-box queries (A) and lowering the precision of confidence scores (B). We report the accuracy (with 95% confidence interval) of our model-less and model-based attacks over 1000 targets models for $n \in \{3,4,5\}$ variables.
• Figure 5: Trade-off between the vulnerability of DP Logistic regression models to our attack (A) and their utility (B). We compute the metrics (with 95% confidence interval) over 1000 target models for $n \in \{3,4,5\}$ variables.