Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Privately Fine-Tuned LLMs Preserve Temporal Dynamics in Tabular Data

Lucas Rosenblatt, Peihan Liu, Ryan McKenna, Natalia Ponomareva

TL;DR

The paper addresses preserving temporal dynamics in differentially private synthetic data for longitudinal tabular records by redefining the privacy unit from individual rows to full user tables. It introduces PATH, a framework that fine-tunes large language models under DP-SGD to autoregressively generate entire user trajectories via a serialization schema, followed by private selection to form a high-utility synthetic corpus. A novel TDCR metric and a comprehensive suite of temporal and distributional evaluations show PATH outperforms marginal-based baselines and non-private prompting in capturing long-range dependencies, state transitions, and manifold coverage across medical, civic, and synthetic datasets. The work demonstrates that private, autoregressive activation of LLMs can yield high-fidelity, temporally coherent synthetic data, enabling safer sharing of sensitive longitudinal records at varying privacy budgets.

Abstract

Research on differentially private synthetic tabular data has largely focused on independent and identically distributed rows where each record corresponds to a unique individual. This perspective neglects the temporal complexity in longitudinal datasets, such as electronic health records, where a user contributes an entire (sub) table of sequential events. While practitioners might attempt to model such data by flattening user histories into high-dimensional vectors for use with standard marginal-based mechanisms, we demonstrate that this strategy is insufficient. Flattening fails to preserve temporal coherence even when it maintains valid marginal distributions. We introduce PATH, a novel generative framework that treats the full table as the unit of synthesis and leverages the autoregressive capabilities of privately fine-tuned large language models. Extensive evaluations show that PATH effectively captures long-range dependencies that traditional methods miss. Empirically, our method reduces the distributional distance to real trajectories by over 60% and reduces state transition errors by nearly 50% compared to leading marginal mechanisms while achieving similar marginal fidelity.

Privately Fine-Tuned LLMs Preserve Temporal Dynamics in Tabular Data

TL;DR

The paper addresses preserving temporal dynamics in differentially private synthetic data for longitudinal tabular records by redefining the privacy unit from individual rows to full user tables. It introduces PATH, a framework that fine-tunes large language models under DP-SGD to autoregressively generate entire user trajectories via a serialization schema, followed by private selection to form a high-utility synthetic corpus. A novel TDCR metric and a comprehensive suite of temporal and distributional evaluations show PATH outperforms marginal-based baselines and non-private prompting in capturing long-range dependencies, state transitions, and manifold coverage across medical, civic, and synthetic datasets. The work demonstrates that private, autoregressive activation of LLMs can yield high-fidelity, temporally coherent synthetic data, enabling safer sharing of sensitive longitudinal records at varying privacy budgets.

Abstract

Research on differentially private synthetic tabular data has largely focused on independent and identically distributed rows where each record corresponds to a unique individual. This perspective neglects the temporal complexity in longitudinal datasets, such as electronic health records, where a user contributes an entire (sub) table of sequential events. While practitioners might attempt to model such data by flattening user histories into high-dimensional vectors for use with standard marginal-based mechanisms, we demonstrate that this strategy is insufficient. Flattening fails to preserve temporal coherence even when it maintains valid marginal distributions. We introduce PATH, a novel generative framework that treats the full table as the unit of synthesis and leverages the autoregressive capabilities of privately fine-tuned large language models. Extensive evaluations show that PATH effectively captures long-range dependencies that traditional methods miss. Empirically, our method reduces the distributional distance to real trajectories by over 60% and reduces state transition errors by nearly 50% compared to leading marginal mechanisms while achieving similar marginal fidelity.
Paper Structure (28 sections, 1 theorem, 21 equations, 24 figures, 12 tables)

This paper contains 28 sections, 1 theorem, 21 equations, 24 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Metrics
  4. "Flattening’’ Longitudinal Data
  5. Proposing the PATH Generative Framework
  6. Baseline Approaches
  7. Experimental Setup
  8. Experimental Results
  9. Related Work
  10. Conclusion
  11. Architecture, Hyperparameters and Privacy Guarantees
  12. Extended Related Work
  13. Data
  14. MIMIC IV
  15. NYC 311 Calls/Requests
  16. ...and 13 more sections

Key Result

Proposition 1

Let $L=3$ and $k=2$. There exists a source distribution supported on a valid set of trajectories $\mathcal{S} \subset \mathcal{Y}$, such that the Maximum Entropy distribution $P^*$ satisfying $2$-local marginal consistency assigns non-zero probability to invalid trajectories $\mathbf{y} \notin \math

Figures (24)

  • Figure 1: Table-wise Distance to Closest Record (TDCR) Distribution Analysis ($\varepsilon=2.0$). We visualize the distribution of distances from generated tables to their nearest neighbor in the real training set for MIMIC-IV Vitalsigns data using our Dynamic Time Warping-based metric. The green shaded area represents the "Real (Held-out)" baseline, showing the distances of real test tables to the training set; a high-fidelity synthetic method should produce a distribution that closely overlaps this baseline. Our proposed method, PATH (Gemma 4B), demonstrates superior overlap with the real data manifold compared to flattened marginal baselines (AIM, Direct) and non-private few-shot prompting (Gemini 2.5 Flash-Lite), which either mode-collapse (tight distribution on the left) or fail to capture the support (too far right).
  • Figure 2: The Flattening Transformation ($\Phi$). Visualizing how a user's longitudinal history (left) is concatenated into a single high-dimensional vector (right). Each timestep becomes a distinct set of attributes (e.g., $A_1^1, A_1^2, \dots$), effectively multiplying the domain dimensionality by the sequence length $L$.
  • Figure 3: The serialization format transforms longitudinal tables into a token sequence. The model learns to predict the next row conditioned on the schema and the history of previous rows.
  • Figure 4: Univariate Density Analysis (Respiratory Rate, MIMIC-IV). We compare the marginal density of respiratory rate distributions across methods at $\varepsilon=2.0$. While marginal-based baselines (AIM, Direct) achieve reasonable Wasserstein scores by broadly covering the support, they suffer from quantization smoothing and noise artifacts that obscure the data's natural shape. In contrast, PATH closely tracks the sharp modal peaks and specific skew of the real distribution (shaded green), demonstrating that PATH can preserve fine-grained univariate trends.
  • Figure 5: HMM Log-Likelihood Distribution Analysis (Synthetic Dataset). We evaluate long-range temporal coherence by scoring generated trajectories against the ground-truth HMM. The real data (shaded green) gives the variance in likelihood scores from our random process. The marginal-based baselines (AIM, Direct) produce a long tail of low-likelihood sequences (shifting left), indicating the generation of temporally incoherent trajectories that violate the underlying state transition logic. While non-private Gemini 2.5 FL (10-shot) produces a tight band of highly likely sequences, it fails to capture the full variance of the distribution (indicative of mode collapse). In contrast, PATH more successfully models the stochastic nature of the true process, producing a likelihood distribution closer to the support of the HMM data.
  • ...and 19 more figures

Theorems & Definitions (6)

  • Definition 1: Flattening Transformation
  • Definition 2: Wasserstein-1 Distance
  • Definition 3: $k$-Local Marginal Consistency
  • Proposition 1: Spurious Trajectories in Local MaxEnt Distributions
  • proof
  • Remark 1