Sample Amplification: Increasing Dataset Size even when Learning is Impossible
Brian Axelrod, Shivam Garg, Vatsal Sharan, Gregory Valiant
TL;DR
The paper investigates how to generate a larger dataset of size $m$ that is indistinguishable from $m$ independent samples from an unknown distribution $D$, given only $n$ samples. It formalizes sample amplification as an amplifier–verifier game and analyzes two natural distribution families: discrete distributions with bounded support and $d$-dimensional Gaussians with unknown mean and fixed covariance. The authors prove tight bounds showing amplification is possible with sublinear input size: for discrete distributions with support size $k$, $m=n+\Theta(n/\sqrt{k})$ can be achieved from $n=O(\sqrt{k})$ samples; for Gaussians with dimension $d$, $m=n+\Theta(n/\sqrt{d})$ can be achieved from $n=\Theta(\sqrt{d})$ samples, with matching lower bounds. Unlike learning, which requires $\Theta(k)$ and $\Theta(d)$ samples respectively, amplification demonstrates that nontrivial dataset expansion is feasible even when learning the distribution is information-theoretically hard. The results illuminate a gap between amplification and learning and offer insights for data augmentation, generative modelling, and downstream analyses, while outlining open directions such as verifer power, instance-optimal schemes, and broader distribution classes.
Abstract
Given data drawn from an unknown distribution, $D$, to what extent is it possible to ``amplify'' this dataset and output an even larger set of samples that appear to have been drawn from $D$? We formalize this question as follows: an $(n,m)$ $\text{amplification procedure}$ takes as input $n$ independent draws from an unknown distribution $D$, and outputs a set of $m > n$ ``samples''. An amplification procedure is valid if no algorithm can distinguish the set of $m$ samples produced by the amplifier from a set of $m$ independent draws from $D$, with probability greater than $2/3$. Perhaps surprisingly, in many settings, a valid amplification procedure exists, even when the size of the input dataset, $n$, is significantly less than what would be necessary to learn $D$ to non-trivial accuracy. Specifically we consider two fundamental settings: the case where $D$ is an arbitrary discrete distribution supported on $\le k$ elements, and the case where $D$ is a $d$-dimensional Gaussian with unknown mean, and fixed covariance. In the first case, we show that an $\left(n, n + Θ(\frac{n}{\sqrt{k}})\right)$ amplifier exists. In particular, given $n=O(\sqrt{k})$ samples from $D$, one can output a set of $m=n+1$ datapoints, whose total variation distance from the distribution of $m$ i.i.d. draws from $D$ is a small constant, despite the fact that one would need quadratically more data, $n=Θ(k)$, to learn $D$ up to small constant total variation distance. In the Gaussian case, we show that an $\left(n,n+Θ(\frac{n}{\sqrt{d}} )\right)$ amplifier exists, even though learning the distribution to small constant total variation distance requires $Θ(d)$ samples. In both the discrete and Gaussian settings, we show that these results are tight, to constant factors. Beyond these results, we formalize a number of curious directions for future research along this vein.