Learning multivariate Gaussians with imperfect advice

TL;DR

The paper tackles distribution learning under imperfect advice by embedding predictions into PAC learning of high-dimensional Gaussians. It introduces two algorithms, TestAndOptimizeMean and TestAndOptimizeCovariance, that adapt sample complexity to the quality of mean and covariance advice via tolerant testing and constrained estimation. The main contributions are explicit, polynomial-time upper bounds that beat standard bounds when advice is good and tight information-theoretic lower bounds showing limits when advice is poor. The work hinges on tolerant testers, LASSO-based mean estimation, and SDP-based covariance estimation to deliver robust, scalable learning with side information. This framework offers practical benefits for scenarios with partial distributional knowledge and supports principled trade-offs between prediction accuracy and data needs.

Abstract

We revisit the problem of distribution learning within the framework of learning-augmented algorithms. In this setting, we explore the scenario where a probability distribution is provided as potentially inaccurate advice on the true, unknown distribution. Our objective is to develop learning algorithms whose sample complexity decreases as the quality of the advice improves, thereby surpassing standard learning lower bounds when the advice is sufficiently accurate. Specifically, we demonstrate that this outcome is achievable for the problem of learning a multivariate Gaussian distribution $N(\boldsymbolμ, \boldsymbolΣ)$ in the PAC learning setting. Classically, in the advice-free setting, $\tildeΘ(d^2/\varepsilon^2)$ samples are sufficient and worst case necessary to learn $d$-dimensional Gaussians up to TV distance $\varepsilon$ with constant probability. When we are additionally given a parameter $\tilde{\boldsymbolΣ}$ as advice, we show that $\tilde{O}(d^{2-β}/\varepsilon^2)$ samples suffices whenever $\| \tilde{\boldsymbolΣ}^{-1/2} \boldsymbolΣ \tilde{\boldsymbolΣ}^{-1/2} - \boldsymbol{I_d} \|_1 \leq \varepsilon d^{1-β}$ (where $\|\cdot\|_1$ denotes the entrywise $\ell_1$ norm) for any $β> 0$, yielding a polynomial improvement over the advice-free setting.

Figures (4)

• Figure 1: Consider partitioning a $d \times d$ matrix (i.e. $d = 5$, $q = 2$) with $w = 4$ blocks $\{(1,2,3), (1,4,5), (2,4,5), (3,4,5)\}$, each of size $k = 3$. Every cell in the original $5 \times 5$ matrix appears in at least $a = 1$ and at most $b = 3$ times across all the induced submatrices.
• Figure 2: Here, $d = 500$, $s = \{100, 200, 300\}$, and $q = \|\bm{\mu} - \widetilde{\bm{\mu}} \|_1 = 50$. Error bars show standard deviation over $10$ runs.
• Figure 3: Here, $d = 500$, $s = 100$, and $q = \|\bm{\mu} - \widetilde{\bm{\mu}} \|_1 \in \{0.1, 20, 30\}$. Error bars show standard deviation over $10$ runs.
• Figure 4: Here, $d = 500$, $s = 100$, and $q = \|\bm{\mu} - \widetilde{\bm{\mu}} \|_1 \in \{0.1, 10, 20, 30, 40, 50, 1000, 10000, 100000\}$. Error bars show standard deviation over $10$ runs. Observe that the slope of the green line looks the same for all $q \geq 1000$ instances.

Theorems & Definitions (84)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Lemma 4: Tolerant mean tester
• Lemma 4: Tolerant covariance tester
• Definition 5: Partitioning scheme
• Lemma 6: Chapter 5.6 of horn2012matrix
• Lemma 6
• Definition 7: Projected vector