Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Reconstruction: Identifiability and Optimization with Sample Splitting

Yujie Shen, Zihan Wang, Jian Qian, Qi Lei

TL;DR

This work introduces sample splitting, a curvature-aware refinement step applicable to general reconstruction objectives (not limited to KKT-based formulations), and discusses the sufficient conditions for KKT system of two-layer networks with polynomial activations to uniquely determine the training data.

Abstract

Training data reconstruction from KKT conditions has shown striking empirical success, yet it remains unclear when the resulting KKT equations have unique solutions and, even in identifiable regimes, how to reliably recover solutions by optimization. This work hereby focuses on these two complementary questions: identifiability and optimization. On the identifiability side, we discuss the sufficient conditions for KKT system of two-layer networks with polynomial activations to uniquely determine the training data, providing a theoretical explanation of when and why reconstruction is possible. On the optimization side, we introduce sample splitting, a curvature-aware refinement step applicable to general reconstruction objectives (not limited to KKT-based formulations): it creates additional descent directions to escape poor stationary points and refine solutions. Experiments demonstrate that augmenting several existing reconstruction methods with sample splitting consistently improves reconstruction performance.

Data Reconstruction: Identifiability and Optimization with Sample Splitting

TL;DR

This work introduces sample splitting, a curvature-aware refinement step applicable to general reconstruction objectives (not limited to KKT-based formulations), and discusses the sufficient conditions for KKT system of two-layer networks with polynomial activations to uniquely determine the training data.

Abstract

Training data reconstruction from KKT conditions has shown striking empirical success, yet it remains unclear when the resulting KKT equations have unique solutions and, even in identifiable regimes, how to reliably recover solutions by optimization. This work hereby focuses on these two complementary questions: identifiability and optimization. On the identifiability side, we discuss the sufficient conditions for KKT system of two-layer networks with polynomial activations to uniquely determine the training data, providing a theoretical explanation of when and why reconstruction is possible. On the optimization side, we introduce sample splitting, a curvature-aware refinement step applicable to general reconstruction objectives (not limited to KKT-based formulations): it creates additional descent directions to escape poor stationary points and refine solutions. Experiments demonstrate that augmenting several existing reconstruction methods with sample splitting consistently improves reconstruction performance.
Paper Structure (42 sections, 6 theorems, 92 equations, 14 figures, 1 algorithm)

This paper contains 42 sections, 6 theorems, 92 equations, 14 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setup
  4. Identifiability of Two-Layer Networks
  5. Linear networks
  6. Homogeneous polynomial activations
  7. Quadratic activation.
  8. Polynomial activation ($\alpha\ge 3$).
  9. Non-homogeneous polynomial activations
  10. Homogenization.
  11. Implicit bias and reconstruction.
  12. The Sample Splitting Algorithm
  13. Beyond KKT: a unified reconstruction objective
  14. Algorithm
  15. Convergence Analysis
  16. ...and 27 more sections

Key Result

Theorem 4.1

Assume $\alpha\ge 3$ and let $(a,W)$ be any KKT point associated with samples $\{(x_i,y_i)\}_{i=1}^n$ and multipliers $\{\lambda_i\}_{i=1}^n$. Suppose the interpolation condition holds: Then $\mathcal{T}$ is uniquely determined by $(a,W)$. Moreover, if we assume $\{x_i\}_{i=1}^n$ are independent and satisfies $\|x_i\|_2=1$, one can recover the active components $\{(x_i,b_i)\}_{i\in S}$ from $\mat

Figures (14)

  • Figure 1: Top 25 images reconstructed from MLP trained on 100 images using loo2024's (row 1), loo2024's with sample splitting (row 2) , and corresponding nearest neighbors from the dataset (row 3).
  • Figure 2: Per-sample comparison of reconstruction metrics before and after splitting using loo2024's method. Each point corresponds to a training sample, with the horizontal axis denoting the metric value without splitting and the vertical axis denoting the metric value with splitting.
  • Figure 3: Reconstruction loss and metrics for haim2022's method with and without sample splitting using 500 training samples, with different initial reconstruction sizes per class.
  • Figure 4: Optimization trajectories of a representative reconstructed sample using buzaglo2024's method, with and without sample splitting.
  • Figure 5: Per-sample metric comparison for CIFAR-10 and MNIST reconstructions using haim2022's method. Each point corresponds to a training sample, with axes denoting metrics before and after splitting.
  • ...and 9 more figures

Theorems & Definitions (13)

  • Theorem 4.1
  • proof : Proof sketch
  • Definition 5.1: Approximate second-order stationarity
  • Theorem 5.1
  • proof : Proof of Theorem \ref{['thm:tensor']}
  • Lemma B.1: Infinitesimal Splitting Expansion
  • proof
  • Theorem B.1: Optimal Infinitesimal Splitting Strategy
  • proof
  • Lemma B.2
  • ...and 3 more