Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compressed Empirical Measures (in finite dimensions)

Steffen Grünewälder

TL;DR

The paper develops a principled approach to compressing the empirical measure in finite-dimensional RKHSs by exploiting the convex geometry of the mean embedding and its convex hull. By bounding the width of the population convex set and ensuring the empirical mean lies well inside it, the authors derive high-probability lower bounds on the size of balls contained in empirical convex sets, enabling coresets for kernel-based inference. They connect these geometric insights to concrete statistical tasks (mean embeddings, covariance, and weighted embeddings) and analyze algorithmic implications for kernel herding and the conditional gradient method, including near-optimal compression rates in several regimes and trade-offs with computation. The work also discusses limitations in infinite dimensions, demonstrates how to adapt the theory to data supported on subsets, and presents simultaneous compression techniques via direct sums and quotient spaces. Collectively, the results offer a rigorous framework for efficient, provable data compression in RKHS-based inference with practical implications for two-sample testing and kernel ridge regression. The methods balance width-based geometric arguments with spectral properties of covariance operators to yield robust probabilistic guarantees for compression performance.}

Abstract

We study approaches for compressing the empirical measure in the context of finite dimensional reproducing kernel Hilbert spaces (RKHSs). In this context, the empirical measure is contained within a natural convex set and can be approximated using convex optimization methods. Such an approximation gives rise to a coreset of data points. A key quantity that controls how large such a coreset has to be is the size of the largest ball around the empirical measure that is contained within the empirical convex set. The bulk of our work is concerned with deriving high probability lower bounds on the size of such a ball under various conditions and in various settings: we show how conditions on the density of the data and the kernel function can be used to infer such lower bounds; we further develop an approach that uses a lower bound on the smallest eigenvalue of a covariance operator to provide lower bounds on the size of such a ball; we extend the approach to approximate covariance operators and we show how it can be used in the context of kernel ridge regression. We also derive compression guarantees when standard algorithms like the conditional gradient method are used and we discuss variations of such algorithms to improve the runtime of these standard algorithms. We conclude with a construction of an infinite dimensional RKHS for which the compression is poor, highlighting some of the difficulties one faces when trying to move to infinite dimensional RKHSs.

Compressed Empirical Measures (in finite dimensions)

TL;DR

The paper develops a principled approach to compressing the empirical measure in finite-dimensional RKHSs by exploiting the convex geometry of the mean embedding and its convex hull. By bounding the width of the population convex set and ensuring the empirical mean lies well inside it, the authors derive high-probability lower bounds on the size of balls contained in empirical convex sets, enabling coresets for kernel-based inference. They connect these geometric insights to concrete statistical tasks (mean embeddings, covariance, and weighted embeddings) and analyze algorithmic implications for kernel herding and the conditional gradient method, including near-optimal compression rates in several regimes and trade-offs with computation. The work also discusses limitations in infinite dimensions, demonstrates how to adapt the theory to data supported on subsets, and presents simultaneous compression techniques via direct sums and quotient spaces. Collectively, the results offer a rigorous framework for efficient, provable data compression in RKHS-based inference with practical implications for two-sample testing and kernel ridge regression. The methods balance width-based geometric arguments with spectral properties of covariance operators to yield robust probabilistic guarantees for compression performance.}

Abstract

We study approaches for compressing the empirical measure in the context of finite dimensional reproducing kernel Hilbert spaces (RKHSs). In this context, the empirical measure is contained within a natural convex set and can be approximated using convex optimization methods. Such an approximation gives rise to a coreset of data points. A key quantity that controls how large such a coreset has to be is the size of the largest ball around the empirical measure that is contained within the empirical convex set. The bulk of our work is concerned with deriving high probability lower bounds on the size of such a ball under various conditions and in various settings: we show how conditions on the density of the data and the kernel function can be used to infer such lower bounds; we further develop an approach that uses a lower bound on the smallest eigenvalue of a covariance operator to provide lower bounds on the size of such a ball; we extend the approach to approximate covariance operators and we show how it can be used in the context of kernel ridge regression. We also derive compression guarantees when standard algorithms like the conditional gradient method are used and we discuss variations of such algorithms to improve the runtime of these standard algorithms. We conclude with a construction of an infinite dimensional RKHS for which the compression is poor, highlighting some of the difficulties one faces when trying to move to infinite dimensional RKHSs.
Paper Structure (75 sections, 21 theorems, 443 equations, 5 figures, 4 algorithms)

This paper contains 75 sections, 21 theorems, 443 equations, 5 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Lower bounding the width of $C$
  3. Locating $\mathfrak{m}$ within $C$
  4. Convergence of $C_n$ to $C$
  5. Assumptions on the spectrum of the covariance operator
  6. Adapting the approach to concrete statistical problems
  7. Simultaneous approximation.
  8. Implications for algorithms
  9. Slow rate of convergence in infinite dimensions
  10. Literature
  11. Preliminaries
  12. Separable processes and Rademacher complexities.
  13. Bochner integrals and $\mathcal{L}^p(\mu,\mathcal{H})$.
  14. Tensor products.
  15. Covariance operators.
  16. ...and 60 more sections

Key Result

Lemma 1

Let $\mathcal{X}$ be a set, $k$ a kernel on $\mathcal{X}$ such that the corresponding RKHS $\mathcal{H}$ is $d$-dimensional. For any $c\in \mathbb{R}$, $\{h : \|h\|_\infty \leq c \}$ is a compact subset of $\mathcal{H}$. Furthermore, for $h \in \mathcal{H}$ and any points $x_1,\ldots,x_d$ for which where $\lambda_d$ is the smallest eigenvalue of the kernel matrix for the points $x_1,\ldots,x_d$.

Figures (5)

  • Figure 1: (i) The figure depicts how a subset or coreset of the sample is selected: the data is embedded in $\mathcal{H}$ by using the kernel function of $\mathcal{H}$. An approximation algorithm is then applied to the convex polytope in $\mathcal{H}$ to find an approximation of $\mathfrak{m}$ that uses only a few extremes of the convex polytope. The pre-images of these extremes are the sample points that are selected as the coreset. (ii) For most statistical problems approximating $\mathfrak{m}$ itself is insufficient and one has to approximate closely related quantities. In the case of least-squares regression, one has to approximate the operator $\mathfrak{C}_{y,n} \in \widehat{\mathcal{H} \odot \mathcal{H}}$ (see Section \ref{['sec:prelim']} and Section \ref{['sec:related_approx']} for the definitions), which is closely related to the empirical covariance operator, and a 'weighted' mean embedding $\mathfrak{m}_{y,n} \in {\mathbb{R}'} \otimes \mathcal{H}$. It is often of interest to approximate $\mathfrak{C}_{y,n}$ and $\mathfrak{m}_{y,n}$ simultaneously, for instance, when building a coreset for least-squares regression. This can be achieved by considering the direct sum $(\widehat{\mathcal{H} \odot \mathcal{H}}) \oplus ({\mathbb{R}'} \otimes \mathcal{H})$ and a 'direct sum' of the convex polytopes in the two spaces. The relation between the extremes of the convex polytopes is highlighted in the figure through the dotted lines; i.e. an algorithm will select a pair that is connected by a dotted line and by selecting such a pair of extremes the approximation of both the covariance and mean element will change.
  • Figure 2: The figure summarizes some of the key questions we address in this paper: (i) This is the central question in this paper; 'how large a ball exists within the empirical convex set $C_n$ around $\mathfrak{m}_n$?' (ii) This question can be addressed by first controlling the width of $C$ itself in different directions $h_1,h_2,\ldots \in \mathcal{H}$. The width in such a direction $h$ is the size of the projection of $C$ on the span of the function $h \in \mathcal{H}$. Lower bounds on the width that hold simultaneously for all relevant $h$ translate to the existence of a ball in $C$; furthermore, the size of the ball is directly related to the lower bounds on the width. (iii) We need not just any ball in $C$ but one that is centered at $\mathfrak{m}$. Now, generally, $\mathfrak{m}$ can lie close to the boundary and no large ball around it might exist. However, under certain natural conditions, it can be ruled out that $\mathfrak{m}$ will lie too closely to the boundary. In particular, under these conditions, we can control the ratio of $a/b$ for the segments shown in the figure. Controlling this ratio for all relevant $h\in \mathcal{H}$ allows us to show that there exists a ball around $\mathfrak{m}$ in $C$. (iv) To translate this back to $C_n$ and $\mathfrak{m}_n$ we are making use of empirical process theory to control the convergence of $C_n \to C$ and $\mathfrak{m}_n \to \mathfrak{m}$ which allows us to lower bound the size of a ball around $\mathfrak{m}_n$ in $C_n$ with high probability. Similarly to (ii) we control the convergence per direction $h$ and then use high probability guarantees that hold simultaneously for all relevant $h$.
  • Figure 3: The figure shows lower bounds on $\left\lvert\left\langle e_n,w\right\rangle\right\rvert$ in dependence of the first element $a_m$ that has not yet been chosen. The shaded area is a lower bound on $\left\lVert w\right\rVert$ when $m= 10^{20}$. The norm of $w$ goes to infinity in $m$ which implies that the kernel herding algorithm converges with a rate that is slower than $1/t$.
  • Figure 4: (i) The figure show $C$ as a subset of $\mathcal{H}$. The diagonal (blue) line is the $\text{\upshape{span}\,}\{u\}$ for some function $u\in \mathcal{H}, \|u\|=1$, The short lines connecting this line to the ellipse indicate the projection of $C$ on $\text{\upshape{span}\,}\{h\}$. In particular, the distance between the two short lines is $\text{\upshape{width}\,}_u(C)$. The long line which is orthogonal to $\text{\upshape{span}\,}\{u\}$ (red) indicates a threshold; the interest is here if $C$ extends past this threshold. (ii) The question if $C$ extends past the threshold is rephrased in this figure by focusing on $\text{\upshape{span}\,}\{u\}$ and considering the probability that values $u(x)$ are attained that lie beyond the threshold. In this figure, we assume for simplicity that the measure on $C$ induces a density function $p(y)$ through the projection on $\text{\upshape{span}\,}\{u\}$, where $y$ goes over the range of $u$. The threshold is in this figure set to $-c$ and $C$ extends past the threshold if the density function is non-zero to the left of $-c$. (iii) To link this construction to the empirical measure we use the function $\psi_\gamma$ whose graph is plotted in this figure against $u(x)$. The motivation is here to appromxiate the indicator function corresponding to the event $u(X) \leq - c$ from below by a continuous function. The parameter $\gamma$ controls the approximation and for $\gamma \to 0$ the function $\psi_\gamma$ converges to the indicator function.
  • Figure 5: The three plots on the left show in blue polynomials of degree $2,3$ and $4$ respectively. The orange lines correspond to the constant functions that are best approximated by these polynomials. The right most plot shows the corresponding approximation error in $\|\cdot\|_\infty$ (orange curve) and our lower bound on the approximation error (blue curve). Note that the approximation error is calculated for the three curves in the left plots and is only an upper bound for the best approximation error that can be attained.

Theorems & Definitions (40)

  • Lemma 1
  • proof
  • Example 1
  • Lemma 2
  • Lemma 3
  • Example 2
  • Proposition 1
  • Example 3
  • Proposition 2
  • proof
  • ...and 30 more