Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coreset selection for the Sinkhorn divergence and generic smooth divergences

Alex Kokot, Alex Luedtke

TL;DR

This work introduces CO2, a framework for lossless data compression with respect to generic smooth divergences by reducing the coreset selection problem to maximum mean discrepancy (MMD) minimization via a second-order Hadamard expansion. Focusing on the Sinkhorn divergence, the authors prove new regularity properties for entropically regularized OT and establish a quadratic expansion centered at the empirical distribution, enabling poly-logarithmic coreset sizes to achieve asymptotically optimal reconstruction. The method leverages Nyström-based kernel approximations and a recombination algorithm to efficiently construct coresets with provable guarantees, bridging coreset selection and kernel quadrature. Empirically, Sinkhorn-CO2 demonstrates strong reconstruction of distributions and label proportions on MNIST and synthetic data, highlighting practical gains in high-dimensional settings and offering a pathway to refine algorithmic efficiency and theoretical bounds. Overall, the paper links coreset theory, kernel methods, and entropic OT to deliver scalable, theoretically grounded coresets for smooth divergences with tangible impact on large-scale data analysis tasks.

Abstract

We introduce CO2, an efficient algorithm to produce convexly-weighted coresets with respect to generic smooth divergences. By employing a functional Taylor expansion, we show a local equivalence between sufficiently regular losses and their second order approximations, reducing the coreset selection problem to maximum mean discrepancy minimization. We apply CO2 to the Sinkhorn divergence, providing a novel sampling procedure that requires poly-logarithmically many data points to match the approximation guarantees of random sampling. To show this, we additionally verify several new regularity properties for entropically regularized optimal transport of independent interest. Our approach leads to a new perspective linking coreset selection and kernel quadrature to classical statistical methods such as moment and score matching. We showcase this method with a practical application of subsampling image data, and highlight key directions to explore for improved algorithmic efficiency and theoretical guarantees.

Coreset selection for the Sinkhorn divergence and generic smooth divergences

TL;DR

This work introduces CO2, a framework for lossless data compression with respect to generic smooth divergences by reducing the coreset selection problem to maximum mean discrepancy (MMD) minimization via a second-order Hadamard expansion. Focusing on the Sinkhorn divergence, the authors prove new regularity properties for entropically regularized OT and establish a quadratic expansion centered at the empirical distribution, enabling poly-logarithmic coreset sizes to achieve asymptotically optimal reconstruction. The method leverages Nyström-based kernel approximations and a recombination algorithm to efficiently construct coresets with provable guarantees, bridging coreset selection and kernel quadrature. Empirically, Sinkhorn-CO2 demonstrates strong reconstruction of distributions and label proportions on MNIST and synthetic data, highlighting practical gains in high-dimensional settings and offering a pathway to refine algorithmic efficiency and theoretical bounds. Overall, the paper links coreset theory, kernel methods, and entropic OT to deliver scalable, theoretically grounded coresets for smooth divergences with tangible impact on large-scale data analysis tasks.

Abstract

We introduce CO2, an efficient algorithm to produce convexly-weighted coresets with respect to generic smooth divergences. By employing a functional Taylor expansion, we show a local equivalence between sufficiently regular losses and their second order approximations, reducing the coreset selection problem to maximum mean discrepancy minimization. We apply CO2 to the Sinkhorn divergence, providing a novel sampling procedure that requires poly-logarithmically many data points to match the approximation guarantees of random sampling. To show this, we additionally verify several new regularity properties for entropically regularized optimal transport of independent interest. Our approach leads to a new perspective linking coreset selection and kernel quadrature to classical statistical methods such as moment and score matching. We showcase this method with a practical application of subsampling image data, and highlight key directions to explore for improved algorithmic efficiency and theoretical guarantees.
Paper Structure (34 sections, 61 theorems, 192 equations, 7 figures, 5 algorithms)

This paper contains 34 sections, 61 theorems, 192 equations, 7 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. The General Case
  3. Notation and definitions
  4. Main results
  5. Selecting $m$
  6. Efficient Kernel Selection
  7. Motivation
  8. The Hadamard Operator
  9. The Remainder
  10. Empirical Estimation
  11. CO2-Recombination Algorithm
  12. Illustration: Sinkhorn Coresets
  13. Overview
  14. Entropic optimal transport potentials
  15. Sinkhorn divergence
  16. ...and 19 more sections

Key Result

Lemma 1

Let $D$ be second order Hadamard differentiable on $\ell^\infty(\mathcal{F}_1)$, where $\mathcal{F}_1$ is the unit ball in an RKHS with characteristic kernel $k$. Then, $\operatorname{MMD}_K(\mathbb{P}_n, P_{m}) = o_p(n^{-1/2})$ implies $D(P_{m}) = O_p(n^{-1})$ and

Figures (7)

  • Figure 1: Eight repetitions of the simulation described in Section \ref{['sec:visualization']}.
  • Figure 2: The reconstruction error $S_\varepsilon(\mathbb{P}_n, P_m)$ in various dimensions (left) and dataset sizes $n$ (right). In the first plot the sample size is fixed at $n=2.5 \times 10^4$, for the latter the dimension is fixed at $d=10$.
  • Figure 3: Percent relative error of $\ddot{S}(\mathbb{P}_n, P_m)$ compared to $S(\mathbb{P}_n, P_m)$ with $P_m$ generated via recombination compression.
  • Figure 4: MMD compression algorithms as well as other sampling methods employed at the secondary stage of our Sinkhorn compression method and the resulting compression error $S(\mathbb{P}_n, P_m)$ induced by these algorithms.
  • Figure 5: PCA coordinates of MNIST data with a Sinkhorn CO2 coreset overlaid on the sample, with figure sizes proportional to the sample weights. For interpretability, only digits 1, 2, and 3 are displayed here.
  • ...and 2 more figures

Theorems & Definitions (61)

  • Lemma 1: Quadratic domination
  • Lemma 2: Fast MMD compression
  • Theorem 3: Main result
  • Lemma 4: Hadamard optimality
  • Lemma 5: Kernel convergence
  • Corollary 6
  • Theorem 7: Sinkhorn compression
  • Theorem 8: Potential differentiation
  • Theorem 9: Sinkhorn kernel
  • Lemma 10: Sinkhorn embedding convergence
  • ...and 51 more