Differentially Private High-dimensional Variable Selection via Integer Programming

TL;DR

This work addresses private sparse variable selection in high dimensions by developing two pure DP estimators for recovering the optimal BSS support, leveraging Mixed Integer Programming to enable scalability to $p$ up to $10^4$. The Top-$R$ and Mistakes mechanisms modify the exponential mechanism to perform structured, scalable sampling over combinatorial supports, with provable recovery guarantees under suitable identifiability and separation conditions. The authors also provide optimization algorithms—MIP reformulations and outer-approximation schemes—that enable efficient computation of the top supports, and extend the framework to hinge loss for sparse classification. Empirically, the proposed methods achieve state-of-the-art support recovery under DP across a wide range of $n$, $p$, $s$, and privacy levels, outperforming existing DP approaches and demonstrating practical impact for privacy-preserving high-dimensional learning.

Abstract

Sparse variable selection improves interpretability and generalization in high-dimensional learning by selecting a small subset of informative features. Recent advances in Mixed Integer Programming (MIP) have enabled solving large-scale non-private sparse regression - known as Best Subset Selection (BSS) - with millions of variables in minutes. However, extending these algorithmic advances to the setting of Differential Privacy (DP) has remained largely unexplored. In this paper, we introduce two new pure differentially private estimators for sparse variable selection, levering modern MIP techniques. Our framework is general and applies broadly to problems like sparse regression or classification, and we provide theoretical support recovery guarantees in the case of BSS. Inspired by the exponential mechanism, we develop structured sampling procedures that efficiently explore the non-convex objective landscape, avoiding the exhaustive combinatorial search in the exponential mechanism. We complement our theoretical findings with extensive numerical experiments, using both least squares and hinge loss for our objective function, and demonstrate that our methods achieve state-of-the-art empirical support recovery, outperforming competing algorithms in settings with up to $p=10^4$.

Figures (14)

• Figure 1: Simulations for $p=10{,}000$, $s=5$, SNR=5, $\rho=0.1$, and $\epsilon=1$ for least squares and hinge loss. The penalty parameter $\lambda$ in Algorithm \ref{['oaalg']} was set to $600$ and $170$ for figures \ref{['p10000fig']} and \ref{['p10000fig-hinge']}, respectively, and the number of MCMC iterations was set to $100{,}000$ for \ref{['p10000fig']}. On the $x$-axis, we vary the value of $n$ and plot the average proportion of draws across 10 independent trials that recovered the right support for each corresponding algorithm. Error bars denote the mean standard error.
• Figure D.1: Numerical experiments for $p=100$ and $p=1{,}000$, with $s=5$, SNR=5, $\rho=0.1$, and $\epsilon=2$. The penalty parameter $\lambda$ in Algorithm \ref{['oaalg']} was set to $120$ and $250$ for figures \ref{['p100fig-testmse']} and \ref{['p1000fig-testmse']}, respectively. On the $x$-axis, we vary the value of $n$ and plot the average test MSE across 10 independent trials. Error bars denote the mean standard error.
• Figure D.2: Numerical experiments for $p=100$ and $p=1{,}000$, with $s=5$, SNR=5, $\rho=0.1$, and $\epsilon=2$. The penalty parameter $\lambda$ in Algorithm \ref{['oaalg']} was set to $120$ and $250$ for figures \ref{['p100fig-utilloss']} and \ref{['p1000fig-utilloss']}, respectively. On the $x$-axis, we vary the value of $n$ and plot the average utility loss across 10 independent trials. Error bars denote the mean standard error.
• Figure D.3: Numerical experiments for $n=1{,}000$, $p=100$, $s=3$, SNR=5, $\rho=0.1$, $\epsilon=1$. Vertical bars denote an objective gap corresponding to an increase in Hamming distance from the optimal support. On the $x$-axis, we vary the value of $R$ and plot the average F1 score and average fraction of correct supports across 10 independent trials for the top-$R$ algorithm. Error bars denote the mean standard error.
• Figure D.4: Numerical experiments for $n=4{,}000$, $p=100$, $s=5$, SNR=5, $\rho=0.1$, $\epsilon=1$. On the $x$-axis, we vary the value of $\lambda$, and in Figures \ref{['abllambdaf1']} and \ref{['abllambdafrac']} and respectively plot the average F1 score and average fraction of correct supports across 10 independent trials for the top-$R$ and mistakes algorithm. Error bars denote the mean standard error.

Theorems & Definitions (34)

• Definition 1: dwork2006differential
• Lemma 1: Exponential Mechanism, exponentialmechanism
• Theorem 1: Privacy for top-$R$ method
• Theorem 2: Privacy for mistakes method
• Remark 1
• Theorem 3: Support recovery for top-$R$ method
• Theorem 4: Support recovery for mistakes method
• Proposition 1
• Remark 2
• Lemma A.1