How Much Training Data is Memorized in Overparameterized Autoencoders? An Inverse Problem Perspective on Memorization Evaluation
Koren Abitbul, Yehuda Dar
TL;DR
The paper tackles memorization in overparameterized autoencoders by recasting it as an inverse problem: recover a training sample $\mathbf{x}$ from a degraded version $\mathbf{y}=\mathbf{H}\mathbf{x}+\boldsymbol{\epsilon}$ using a regularizer $s_f$ induced by the trained autoencoder $f$, while the degradation operator $\mathbf{H}$ is unknown. It introduces a practical alternating-minimization algorithm that embeds a black-box autoencoder inside a plug-and-play-ADMM framework, jointly estimating $\mathbf{x}$ and $\mathbf{H}$ and employing an ADMM inner loop with a proximal-like step that can be replaced by $f$. The authors establish theoretical connections showing certain 2-layer tied autoencoders are Moreau proximal mappings, enabling the plug-and-play substitution, and demonstrate substantial empirical gains in training-data recovery over prior memorization-evaluation methods across multiple architectures and datasets, including moderate overfitting and large-scale regimes. The results offer a scalable, data-specific approach to quantify memorization in autoencoders, with implications for understanding overparameterization, privacy risks, and data-regression capabilities in deep representations. The method improves recoveries on training data while preserving lack of recovery on non-training data, underscoring its role as a precise diagnostic for memorization phenomena in deep autoencoders.
Abstract
Overparameterized autoencoder models often memorize their training data. For image data, memorization is often examined by using the trained autoencoder to recover missing regions in its training images (that were used only in their complete forms in the training). In this paper, we propose an inverse problem perspective for the study of memorization. Given a degraded training image, we define the recovery of the original training image as an inverse problem and formulate it as an optimization task. In our inverse problem, we use the trained autoencoder to implicitly define a regularizer for the particular training dataset that we aim to retrieve from. We develop the intricate optimization task into a practical method that iteratively applies the trained autoencoder and relatively simple computations that estimate and address the unknown degradation operator. We evaluate our method for blind inpainting where the goal is to recover training images from degradation of many missing pixels in an unknown pattern. We examine various deep autoencoder architectures, such as fully connected and U-Net (with various nonlinearities and at diverse train loss values), and show that our method significantly outperforms previous memorization-evaluation methods that recover training data from autoencoders. Importantly, our method greatly improves the recovery performance also in settings that were previously considered highly challenging, and even impractical, for such recovery and memorization evaluation.