Improved Active Learning via Dependent Leverage Score Sampling

TL;DR

The paper tackles efficient active linear regression in the agnostic setting by fusing leverage-score-based marginal sampling with spatially aware pivotal sampling. It introduces a two-step, spatially structured sampling scheme that maintains leverage-score marginals while ensuring wide coverage via a binary-tree pivotal process, leading to up to 50% fewer labeled samples than independent leverage-score sampling in practice. The authors prove a general bound under one-sided $\ell_{\infty}$-independence that matches conventional leverage-score sampling and provide a tighter $O(d/\epsilon)$ bound for polynomial regression, supported by experiments on PDE-inspired test problems. This approach enables more efficient learning of surrogates for parametric PDEs and uncertainty quantification tasks, with strong theoretical and empirical support for spatially distributed sampling.

Abstract

We show how to obtain improved active learning methods in the agnostic (adversarial noise) setting by combining marginal leverage score sampling with non-independent sampling strategies that promote spatial coverage. In particular, we propose an easily implemented method based on the \emph{pivotal sampling algorithm}, which we test on problems motivated by learning-based methods for parametric PDEs and uncertainty quantification. In comparison to independent sampling, our method reduces the number of samples needed to reach a given target accuracy by up to $50\%$. We support our findings with two theoretical results. First, we show that any non-independent leverage score sampling method that obeys a weak \emph{one-sided $\ell_{\infty}$ independence condition} (which includes pivotal sampling) can actively learn $d$ dimensional linear functions with $O(d\log d)$ samples, matching independent sampling. This result extends recent work on matrix Chernoff bounds under $\ell_{\infty}$ independence, and may be of interest for analyzing other sampling strategies beyond pivotal sampling. Second, we show that, for the important case of polynomial regression, our pivotal method obtains an improved bound on $O(d)$ samples.

Figures (11)

• Figure 1: Polynomial approximations to the maximum displacement of a damped harmonic oscillator, as a function of driving frequency and spring constant. (a) is the target value, and samples can be obtained through the numerical solution of a differential equation governing the oscillator. Both (b) and (c) draw $250$ samples using leverage score sampling and perform polynomial regression of degree $20$. (b) uses Bernoulli sampling while (c) uses our pivotal sampling method. Our method gives a better approximation, avoiding artifacts that result from gaps between the Bernoulli samples.
• Figure 2: The results of three different active learning methods used to collect samples to fit a polynomial over $[-1,1]\times [-1,1]$. The image on the left was obtained by collecting points independently at random with probability according to their statistical leverage scores. The image on the right was obtained by collecting samples at the 2-dimensional Chebyshev nodes. The image in the middle shows our method, which collects samples according to leverage scores, but using a non-independent pivotal sampling strategy that ensures samples are evenly spread in spatially.
• Figure 3: Visualization of a binary tree constructed via our Algorithm \ref{['algo:btree']} using the PCA method for a matrix $\mathbf{X}\in\mathbb{R}^{n\times 2}$ containing points on a uniform square grid. For each depth, data points are given the same color if they compose a subtree with root at that depth. As we can see, the method produces uniform recursive spatial partitions, which encourage spatially separated samples.
• Figure 4: Results for active polynomial regression for the damped harmonic oscillator QoI, the heat equation QoI, and the surface reaction model with polynomials varying degree. Our leverage-score based pivotal method outperforms standard Bernoulli leverage score sampling, suggesting the benefits of spatially-aware sampling.
• Figure 5: Polynomial approximation to the maximum temperature of a heat diffusion problem, as a function of time and starting condition. (a) is the target value and both (b) and (c) draw $240$ samples using the leverage score and perform polynomial regression of degree $20$. However, (b) uses Bernoulli sampling while (c) employs our PCA-based pivotal sampling.

Theorems & Definitions (27)

• Definition 1.1: Leverage Score
• Theorem 1.1
• Corollary 1.1
• Theorem 1.2
• Definition 3.1: One-sided $\ell_{\infty}$-independence
• Definition B.1: Homogeneity
• Definition B.1: One-sided $\ell_{\infty}$-independence
• proof : Proof of Theorem \ref{['thm:main']}
• Lemma B.1: Matrix Chernoff for $\ell_{\infty}$-independent Distributions
• proof : Proof of Corollary \ref{['corr:main']}