Do You Really Need Public Data? Surrogate Public Data for Differential Privacy on Tabular Data
Shlomi Hod, Lucas Rosenblatt, Julia Stoyanovich
TL;DR
The paper tackles the challenge of scarce public tabular data for differential privacy by introducing surrogate public data generated solely from dataset schemas using large language models. It develops two generation methods—direct CSV record creation and an agent-based structural causal model approach—to encode plausible variable dependencies without accessing sensitive data. A comprehensive evaluation framework across three DP auxiliary tasks (pretraining, hyperparameter tuning, and privacy–utility estimation) and three real-world datasets (ACS, EDAD, WE) demonstrates that surrogate public data can meaningfully support DP classifier pretraining and offer guidance for DP hyperparameter tuning, especially in low-data settings, though traditional public data often remains superior for privacy–utility estimation. The findings underscore the potential of schema-driven LLM-guided surrogates to alleviate tabular data scarcity in DP while highlighting limitations related to memorization, biases, and the need for robust similarity metrics to predict surrogate usefulness in practice.
Abstract
Differentially private (DP) machine learning often relies on the availability of public data for tasks like privacy-utility trade-off estimation, hyperparameter tuning, and pretraining. While public data assumptions may be reasonable in text and image domains, they are less likely to hold for tabular data due to tabular data heterogeneity across domains. We propose leveraging powerful priors to address this limitation; specifically, we synthesize realistic tabular data directly from schema-level specifications - such as variable names, types, and permissible ranges - without ever accessing sensitive records. To that end, this work introduces the notion of "surrogate" public data - datasets generated independently of sensitive data, which consume no privacy loss budget and are constructed solely from publicly available schema or metadata. Surrogate public data are intended to encode plausible statistical assumptions (informed by publicly available information) into a dataset with many downstream uses in private mechanisms. We automate the process of generating surrogate public data with large language models (LLMs); in particular, we propose two methods: direct record generation as CSV files, and automated structural causal model (SCM) construction for sampling records. Through extensive experiments, we demonstrate that surrogate public tabular data can effectively replace traditional public data when pretraining differentially private tabular classifiers. To a lesser extent, surrogate public data are also useful for hyperparameter tuning of DP synthetic data generators, and for estimating the privacy-utility tradeoff.