Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Turnstile $\ell_p$ leverage score sampling with applications

Alexander Munteanu, Simon Omlor

TL;DR

The paper addresses sampling rows of a dynamically updated matrix in turnstile streams to approximate $\ell_p$ leverage scores. It introduces an $L_{p,p}$ sampler for $p\in[1,2]$ built on thresholded CountSketch and harmonic random rescaling, followed by a conditioning-based postprocessing step to yield a true $\ell_p$-leverage-score sampler. The approach enables $\varepsilon$-coresets for a broad class of regression losses, including logistic regression, with fully polynomial sketch sizes and space, circumventing prior lower-bound limitations via postprocessing of oblivious sketches. Empirical results show the method bridges the gap between oblivious sketching and offline sampling, offering practical benefits for turnstile data processing and large-scale regression. The work opens avenues toward constructing $\ell_p$ spanning sets in turnstile streams and further improving efficiency via sparse embeddings, with broad applicability to core-set constructions in dynamic data settings.

Abstract

The turnstile data stream model offers the most flexible framework where data can be manipulated dynamically, i.e., rows, columns, and even single entries of an input matrix can be added, deleted, or updated multiple times in a data stream. We develop a novel algorithm for sampling rows $a_i$ of a matrix $A\in\mathbb{R}^{n\times d}$, proportional to their $\ell_p$ norm, when $A$ is presented in a turnstile data stream. Our algorithm not only returns the set of sampled row indexes, it also returns slightly perturbed rows $\tilde{a}_i \approx a_i$, and approximates their sampling probabilities up to $\varepsilon$ relative error. When combined with preconditioning techniques, our algorithm extends to $\ell_p$ leverage score sampling over turnstile data streams. With these properties in place, it allows us to simulate subsampling constructions of coresets for important regression problems to operate over turnstile data streams with very little overhead compared to their respective off-line subsampling algorithms. For logistic regression, our framework yields the first algorithm that achieves a $(1+\varepsilon)$ approximation and works in a turnstile data stream using polynomial sketch/subsample size, improving over $O(1)$ approximations, or $\exp(1/\varepsilon)$ sketch size of previous work. We compare experimentally to plain oblivious sketching and plain leverage score sampling algorithms for $\ell_p$ and logistic regression.

Turnstile $\ell_p$ leverage score sampling with applications

TL;DR

The paper addresses sampling rows of a dynamically updated matrix in turnstile streams to approximate leverage scores. It introduces an sampler for built on thresholded CountSketch and harmonic random rescaling, followed by a conditioning-based postprocessing step to yield a true -leverage-score sampler. The approach enables -coresets for a broad class of regression losses, including logistic regression, with fully polynomial sketch sizes and space, circumventing prior lower-bound limitations via postprocessing of oblivious sketches. Empirical results show the method bridges the gap between oblivious sketching and offline sampling, offering practical benefits for turnstile data processing and large-scale regression. The work opens avenues toward constructing spanning sets in turnstile streams and further improving efficiency via sparse embeddings, with broad applicability to core-set constructions in dynamic data settings.

Abstract

The turnstile data stream model offers the most flexible framework where data can be manipulated dynamically, i.e., rows, columns, and even single entries of an input matrix can be added, deleted, or updated multiple times in a data stream. We develop a novel algorithm for sampling rows of a matrix , proportional to their norm, when is presented in a turnstile data stream. Our algorithm not only returns the set of sampled row indexes, it also returns slightly perturbed rows , and approximates their sampling probabilities up to relative error. When combined with preconditioning techniques, our algorithm extends to leverage score sampling over turnstile data streams. With these properties in place, it allows us to simulate subsampling constructions of coresets for important regression problems to operate over turnstile data streams with very little overhead compared to their respective off-line subsampling algorithms. For logistic regression, our framework yields the first algorithm that achieves a approximation and works in a turnstile data stream using polynomial sketch/subsample size, improving over approximations, or sketch size of previous work. We compare experimentally to plain oblivious sketching and plain leverage score sampling algorithms for and logistic regression.
Paper Structure (32 sections, 24 theorems, 83 equations, 4 figures, 2 algorithms)

This paper contains 32 sections, 24 theorems, 83 equations, 4 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Comparison to related work
  4. Algorithms and technical overview
  5. Idea 1: thresholding the CountSketch
  6. Idea 2: controlling random rescaling by means of the harmonic series
  7. Sublinear space with logarithmic overhead
  8. Applications
  9. Idea 3: sampling from mixture distributions and lp conditioning
  10. Idea 4: robustness of various loss functions under small perturbations
  11. Experimental illustration
  12. Conclusion
  13. Preliminaries
  14. The algorithms
  15. Outline of the analysis
  16. ...and 17 more sections

Key Result

Theorem 2.1

Let $\varepsilon,\delta\in(0,1/20],\gamma\in(0,1)$. Let $L$ be the list of tuples in the output of alg:findhh. Further let $S_R(r/20)$ be the subset of rows excluding the $r/20$ largest $\ell_p$ norms and let $M=\sum_{i \in S_R} \lVert a_i\rVert_p^p$. If $r = 8\gamma^{-1} \cdot (12/\varepsilon)^p$ a

Figures (4)

  • Figure 1: Comparison of the approximation ratios for logistic regression, and $\ell_1$ regression on various real-world datasets. The new turnstile data stream sampler (orange) is compared to plain leverage score sampling (red), and to plain oblivious sketching (blue). The plots indicate the median of approximation ratios taken over 21 repetitions for each reduced size. Best viewed in colors, lower is better.
  • Figure 2: Comparison of the approximation ratios and running times for logistic regression on various real-world datasets. The new turnstile data stream sampler for $p=1$ (orange) and a mixture $p=1,q=2$ (lime) is compared to plain leverage score sampling (red), and to plain oblivious sketching (blue). The plots indicate the median of approximation ratios taken over 21 repetitions for each reduced size. Best viewed in colors, lower is better.
  • Figure 3: Comparison of the approximation ratios and running times for $\ell_1$ regression on various real-world datasets. The new turnstile data stream sampler for $p=1$ (orange) and a mixture $p=1,q=2$ (lime) is compared to plain leverage score sampling (red), and to plain oblivious sketching (blue). The plots indicate the median of approximation ratios taken over 21 repetitions for each reduced size. Best viewed in colors, lower is better.
  • Figure 4: Comparison of the approximation ratios and running times for $\ell_{1.5}$ regression on various real-world datasets. The new turnstile data stream sampler for $p=1.5$ (orange) is compared to plain leverage score sampling for $p=1.5$ (red). The plots indicate the median of approximation ratios taken over 21 repetitions for each reduced size. Best viewed in colors, lower is better.

Theorems & Definitions (46)

  • Definition 1.1: $L_{p,p}$ sampling
  • Theorem 2.1
  • Theorem 2.2
  • Proposition 2.3: Nisan92, cf. JayaramWZ22
  • Corollary 3.1
  • Definition 3.2: DasguptaDHKM09
  • Proposition 3.3
  • Theorem 3.4
  • Lemma D.1
  • proof
  • ...and 36 more