Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When fractional quasi p-norms concentrate

Ivan Y. Tyukin, Bogdan Grechuk, Evgeny M. Mirkes, Alexander N. Gorban

TL;DR

This work resolves a long-standing question about the concentration of fractional quasi-norm distances in high dimensions by proving that, for a broad class of i.i.d. distributions without mass at zero, $\ell^p$ quasi-norms with $p\in(0,1)$ exhibit exponential concentration that is uniform in $p$. It also shows that there exist distribution families with mass near zero for which the concentration can be mitigated by choosing $p$ appropriately, and, moreover, that anti-concentration can occur in arbitrarily small neighborhoods of many zero-mass-free distributions. The results hinge on a Chernoff-based large-deviation framework, yielding explicit rate functions $\Lambda^\pm(p,\delta)$ and uniform bounds $f_*^\pm(\delta)$, and they emphasize the role of zero-mass and data encoding in determining concentration behavior. Practically, the findings clarify when fractional quasi-norms may help or fail to alleviate the curse of dimensionality and highlight encoding strategies and preprocessing as levers to influence distance concentration in high-dimensional data analysis.

Abstract

Concentration of distances in high dimension is an important factor for the development and design of stable and reliable data analysis algorithms. In this paper, we address the fundamental long-standing question about the concentration of distances in high dimension for fractional quasi $p$-norms, $p\in(0,1)$. The topic has been at the centre of various theoretical and empirical controversies. Here we, for the first time, identify conditions when fractional quasi $p$-norms concentrate and when they don't. We show that contrary to some earlier suggestions, for broad classes of distributions, fractional quasi $p$-norms admit exponential and uniform in $p$ concentration bounds. For these distributions, the results effectively rule out previously proposed approaches to alleviate concentration by "optimal" setting the values of $p$ in $(0,1)$. At the same time, we specify conditions and the corresponding families of distributions for which one can still control concentration rates by appropriate choices of $p$. We also show that in an arbitrarily small vicinity of a distribution from a large class of distributions for which uniform concentration occurs, there are uncountably many other distributions featuring anti-concentration properties. Importantly, this behavior enables devising relevant data encoding or representation schemes favouring or discouraging distance concentration. The results shed new light on this long-standing problem and resolve the tension around the topic in both theory and empirical evidence reported in the literature.

When fractional quasi p-norms concentrate

TL;DR

This work resolves a long-standing question about the concentration of fractional quasi-norm distances in high dimensions by proving that, for a broad class of i.i.d. distributions without mass at zero, quasi-norms with exhibit exponential concentration that is uniform in . It also shows that there exist distribution families with mass near zero for which the concentration can be mitigated by choosing appropriately, and, moreover, that anti-concentration can occur in arbitrarily small neighborhoods of many zero-mass-free distributions. The results hinge on a Chernoff-based large-deviation framework, yielding explicit rate functions and uniform bounds , and they emphasize the role of zero-mass and data encoding in determining concentration behavior. Practically, the findings clarify when fractional quasi-norms may help or fail to alleviate the curse of dimensionality and highlight encoding strategies and preprocessing as levers to influence distance concentration in high-dimensional data analysis.

Abstract

Concentration of distances in high dimension is an important factor for the development and design of stable and reliable data analysis algorithms. In this paper, we address the fundamental long-standing question about the concentration of distances in high dimension for fractional quasi -norms, . The topic has been at the centre of various theoretical and empirical controversies. Here we, for the first time, identify conditions when fractional quasi -norms concentrate and when they don't. We show that contrary to some earlier suggestions, for broad classes of distributions, fractional quasi -norms admit exponential and uniform in concentration bounds. For these distributions, the results effectively rule out previously proposed approaches to alleviate concentration by "optimal" setting the values of in . At the same time, we specify conditions and the corresponding families of distributions for which one can still control concentration rates by appropriate choices of . We also show that in an arbitrarily small vicinity of a distribution from a large class of distributions for which uniform concentration occurs, there are uncountably many other distributions featuring anti-concentration properties. Importantly, this behavior enables devising relevant data encoding or representation schemes favouring or discouraging distance concentration. The results shed new light on this long-standing problem and resolve the tension around the topic in both theory and empirical evidence reported in the literature.

Paper Structure

This paper contains 22 sections, 15 theorems, 149 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Motivation for fractional $\ell^p$, $p\in(0,1)$ quasi-norms
  3. Contribution and structure of this work
  4. Main assumptions and the general estimate
  5. Exponential concentration of fractional and quasi $p$-norms
  6. Non-uniformity of exponential concentration and mitigating quasi-norm concentration by the choice of $p$
  7. Uniform exponential concentration
  8. Concentration of p-norm for small p
  9. General case
  10. Concentration of quasi p-norms for some common distributions of $x_i$ (small p)
  11. Concentration of p-norm for large p for bounded distributions
  12. The uniform concentration for all p
  13. Concentration of p-norm for small $\delta$
  14. Case 1: any $p>0$ and small $\delta$
  15. Case 2: small $p$ and small $\delta$
  16. ...and 7 more sections

Key Result

Theorem 1

Suppose that Assumptions assume:a and assume:b hold. Then, for any $\delta\in(0,1)$ and any $p\in(0,p_0]$ where $\Lambda^+(p,\delta), \Lambda^-(p,\delta)>0$ and are defined as: The constants $\Lambda^+(p,\delta), \Lambda^-(p,\delta)$ are best possible in the sense that

Figures (4)

  • Figure 1: Empirical probabilities ${\mathbb P}\left(1-\delta\leq\frac{||{\bf x}||_p}{(n\mu_p)^{1/p}} \leq 1+\delta \right)$ as functions of $p$ and $n$ for $\delta=0.1$, computed for vectors sampled from the product distribution satisfying (\ref{['eq:example_concentration_breaks']}) with $a=1/2$, $r=1$. The size of data samples used to compute these probabilities was equal to $5\times 10^5$.
  • Figure 2: Empirical probabilities ${\mathbb P}\left(1-\delta\leq\frac{||{\bf x}||_p}{(n\mu_p)^{1/p}} \leq 1+\delta \right)$ as functions of $p$ and $n$ for $\delta=0.1$, computed for vectors sampled from the product distribution satisfying (\ref{['eq:example_concentration']}) with $a=1/2$, $r=1$, and $\xi=0.001$. The size of data samples used to compute these probabilities was equal to $5\times 10^5$.
  • Figure 3: Top panel: Empirical probabilities ${\mathbb P}\left(\frac{|||{\bf x}_1||_p-||{\bf x}_2||_p|}{(n\mu_p)^{1/p}}<\delta \right)$ computed for ${\bf x}_1, \ {\bf x}_2$ sampled from $[0,1]^n$, $\delta = 0.1$; the sample size was equal to $5\times 10^5$. Bottom panel: Empirical probabilities ${\mathbb P}\left(1-\delta\leq\frac{||{\bf x}||_p}{(n\mu_p)^{1/p}} \leq 1+\delta \right)$ as functions of $p$ and $n$ for $\delta=0.1$, computed for vectors sampled from the uniform distribution in $[0,1]^n$. The size of data samples used to compute these probabilities was equal to $5\times 10^5$.
  • Figure 4: Theoretical exponential concentration rates for the uniform distribution in $[-1,1]^n$. Left panel shows $\Lambda^-(p,\delta)$ for $p=0.01, 0.1, 0.5, 1, 2$. Right panel shows $\Lambda^+(p,\delta)$ for $p=0.01, 0.1, 0.5, 1, 2$ (see Theorem \ref{['theorem:p-norm-concentration']}). Red solid line in both panels shows the bound stemming from Proposition \ref{['prop:pdepla0']}, and which is independent on the values of $p$ (see (\ref{['eq:exp_bound_uniform']})).

Theorems & Definitions (17)

  • Theorem 1
  • Proposition 1
  • Proposition 2
  • Theorem 2
  • Remark 1
  • Proposition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Theorem 3
  • ...and 7 more