Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the use of the M-quantiles for outlier detection in multivariate data

Sajal Chakroborty, Ram Iyer, A. Alexandre Trindade

TL;DR

This work critically examines M-quantiles built from the Koltchinskii K-function for multivariate outlier detection. It shows that in odd dimensions $d\ge 3$, the density is tied to a poly-Laplacian of the K-function rather than its gradient, undermining the view that the K-transform yields true multivariate quantiles; in $\mathbb{R}^2$ a zoom-in effect distorts density when mapping to the unit disk, complicating outlier assessment. The authors provide an MM-based algorithm to compute geometric quantiles and illustrate pathological behavior with nonstandard distributions, challenging the use of inverse K-transform for outlier detection in higher dimensions. Collectively, the results motivate dimension-dependent caution and point toward alternative center-outward transport approaches for multivariate quantiles.

Abstract

Defining a successful notion of a multivariate quantile has been an open problem for more than half a century, motivating a plethora of possible solutions. Of these, the approach of [8] and [25] leading to M-quantiles, is very appealing for its mathematical elegance combining elements of convex analysis and probability theory. The key idea is the description of a convex function (the K-function) whose gradient (the K-transform) is in one-to-one correspondence between all of R^d and the unit ball in R^d. By analogy with the d=1 case where the K-transform is a cumulative distribution function-like object (an M-distribution), the fact that its inverse is guaranteed to exist lends itself naturally to providing the basis for the definition of a quantile function for all d>=1. Over the past twenty years the resulting M-quantiles have seen applications in a variety of fields, primarily for the purpose of detecting outliers in multidimensional spaces. In this article we prove that for odd d>=3, it is not the gradient but a poly-Laplacian of the K-function that is (almost everywhere) proportional to the density function. For d even one cannot establish a differential equation connecting the K-function with the density. These results show that usage of the K-transform for outlier detection in higher odd-dimensions is in principle flawed, as the K-transform does not originate from inversion of a true M-distribution. We demonstrate these conclusions in two dimensions through examples from non-standard asymmetric distributions. Our examples illustrate a feature of the K-transform whereby regions in the domain with higher density map to larger volumes in the co-domain, thereby producing a magnification effect that moves inliers closer to the boundary of the co-domain than outliers. This feature obviously disrupts any outlier detection mechanism that relies on the inverse K-transform.

On the use of the M-quantiles for outlier detection in multivariate data

TL;DR

This work critically examines M-quantiles built from the Koltchinskii K-function for multivariate outlier detection. It shows that in odd dimensions , the density is tied to a poly-Laplacian of the K-function rather than its gradient, undermining the view that the K-transform yields true multivariate quantiles; in a zoom-in effect distorts density when mapping to the unit disk, complicating outlier assessment. The authors provide an MM-based algorithm to compute geometric quantiles and illustrate pathological behavior with nonstandard distributions, challenging the use of inverse K-transform for outlier detection in higher dimensions. Collectively, the results motivate dimension-dependent caution and point toward alternative center-outward transport approaches for multivariate quantiles.

Abstract

Defining a successful notion of a multivariate quantile has been an open problem for more than half a century, motivating a plethora of possible solutions. Of these, the approach of [8] and [25] leading to M-quantiles, is very appealing for its mathematical elegance combining elements of convex analysis and probability theory. The key idea is the description of a convex function (the K-function) whose gradient (the K-transform) is in one-to-one correspondence between all of R^d and the unit ball in R^d. By analogy with the d=1 case where the K-transform is a cumulative distribution function-like object (an M-distribution), the fact that its inverse is guaranteed to exist lends itself naturally to providing the basis for the definition of a quantile function for all d>=1. Over the past twenty years the resulting M-quantiles have seen applications in a variety of fields, primarily for the purpose of detecting outliers in multidimensional spaces. In this article we prove that for odd d>=3, it is not the gradient but a poly-Laplacian of the K-function that is (almost everywhere) proportional to the density function. For d even one cannot establish a differential equation connecting the K-function with the density. These results show that usage of the K-transform for outlier detection in higher odd-dimensions is in principle flawed, as the K-transform does not originate from inversion of a true M-distribution. We demonstrate these conclusions in two dimensions through examples from non-standard asymmetric distributions. Our examples illustrate a feature of the K-transform whereby regions in the domain with higher density map to larger volumes in the co-domain, thereby producing a magnification effect that moves inliers closer to the boundary of the co-domain than outliers. This feature obviously disrupts any outlier detection mechanism that relies on the inverse K-transform.
Paper Structure (13 sections, 7 theorems, 86 equations, 4 figures)

This paper contains 13 sections, 7 theorems, 86 equations, 4 figures.

Table of Contents

  1. Introduction
  2. The K-transform: Definition and Outlier Detection
  3. The K-transform in Relation to the Distribution
  4. The zoom-in effect for $\mathbb{R}^2$
  5. Conclusion
  6. Algorithm for Computation of Geometric Quantiles
  7. Proof of Theorem \ref{['th:Leibniz rule']}
  8. Proof of Lemma \ref{['Lemma 1']}
  9. Proof of Lemma \ref{['Lemma 2']}
  10. Proof of Lemma \ref{['Lemma 3']}
  11. Proof of Lemma \ref{['Lemma 4']}
  12. Proof of Theorem \ref{['Theorem']}
  13. Proof of Corollary \ref{['corollary']}

Key Result

Theorem 2.1

Suppose $f(s,x)$ is bounded for all $x\in [a,b]$, and is both bounded and integrable with respect to Lebesgue measure $\lambda$ for all $s\in [\alpha,\beta]$. For any Lipschitz continuous functions $g,h:[\alpha,\beta] \rightarrow [a,b]$, let: For all $s \in (\alpha,\beta)$ and for all $x \in [a,b] \setminus E_0,$ where $\lambda(E_0) = 0$, let $\partial f(s,x)/\partial s$ be defined on $(\alpha,\b

Figures (4)

  • Figure 1: K-transform and its inverse for a bivariate Gaussian with zero mean, unit standard deviation, and correlation coefficient of $0.75$. The K-transform is shown in Figure \ref{['fig:K-transform']}, and the inverse K-transform in Figure \ref{['fig:Inverse K-outliers']}, along with 3000 data points simulated from the distribution.
  • Figure 2: Banana-shaped distribution with $n=20,000$ sample points hallin-etal-2021. Concentric circles of radius 0.50 (blue), 0.75 (red), and 0.90 (maroon) are shown in Figure \ref{['fig:Inverse-K-Transform (BananaShape)']}. Their corresponding K-transforms are in Figure \ref{['fig:K-Transform (BananaShape)']}. This example illustrates the zoom-in effect of the K-transform. Points outside the support of the distribution in the domain occupy a small region close to the boundary of the unit circle in the co-domain. Hence, $1-\|\nabla f\|_2$ cannot be interpreted as a depth function.
  • Figure 3: Spiral-shaped distribution with $n=2259$ sample points xie2020local. Figure \ref{['fig:K-transform(Spiral)']} shows the K-transform and \ref{['fig:Inverse-K-transform(Spiral)']} shows the inverse K-transform. This example illustrates zoom-in effect and the low power of the K-transform method. Points in the empty spaces are within the $r=0.5$ M-quantile while points in the outer spiral of the data are outside the circle of radius $0.9$ in the co-domain.
  • Figure 4: This example illustrates zoom-in effect and the low power of the K-transform method for a non-standard square-shaped distribution with $n=1242$ sample points xie2020local. Figure \ref{['fig:K-transform(SquareShape)']} shows the K-transform and \ref{['fig:Inverse-K-transform(SquareShape)']} shows the inverse K-transform. Points in the empty spaces within the $r=0.5$ M-quantile should be outliers but are not classified as such by the method.

Theorems & Definitions (14)

  • Theorem 2.1: Lebesgue Measure Leibniz Rule
  • Remark 3.1
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Lemma 3.5
  • Theorem 3.6
  • corollary 1
  • proof
  • proof
  • ...and 4 more