Capturing the Denoising Effect of PCA via Compression Ratio
Chandra Sekhar Mukherjee, Nikhil Doerkar, Jiapeng Zhang
TL;DR
The paper introduces compression ratio, a geometry-based metric $\Delta_{X,k'}(i,i') = \frac{\|\boldsymbol{x_i}-\boldsymbol{x_{i'}}\|}{\|\Pi^{k'}_X(\boldsymbol{x_i}-\boldsymbol{x_{i'}})\|}$, to quantify PCA’s denoising effect on high-dimensional data with underlying community structure. It analyzes a random vector mixture model to bound intra- and inter-community compression under PCA, linking these bounds to improved downstream clustering and guiding an outlier-detection approach based on the variance of compression ratios. The authors extend the model to include outliers, provide theoretical justification, and compare against standard detectors, showing competitive performance, especially in noisy regimes. They validate the framework on real-world single-cell RNA-seq datasets, demonstrating that intra-community compression consistently exceeds inter-community compression and that removing outliers improves PCA+KMeans clustering performance, often outperforming baselines. Overall, the work provides a principled, algorithmically simple method to quantify PCA’s denoising and leverage it to improve clustering in high-dimensional, noisy data, with practical impact in domains like single-cell genomics.
Abstract
Principal component analysis (PCA) is one of the most fundamental tools in machine learning with broad use as a dimensionality reduction and denoising tool. In the later setting, while PCA is known to be effective at subspace recovery and is proven to aid clustering algorithms in some specific settings, its improvement of noisy data is still not well quantified in general. In this paper, we propose a novel metric called \emph{compression ratio} to capture the effect of PCA on high-dimensional noisy data. We show that, for data with \emph{underlying community structure}, PCA significantly reduces the distance of data points belonging to the same community while reducing inter-community distance relatively mildly. We explain this phenomenon through both theoretical proofs and experiments on real-world data. Building on this new metric, we design a straightforward algorithm that could be used to detect outliers. Roughly speaking, we argue that points that have a \emph{lower variance of compression ratio} do not share a \emph{common signal} with others (hence could be considered outliers). We provide theoretical justification for this simple outlier detection algorithm and use simulations to demonstrate that our method is competitive with popular outlier detection tools. Finally, we run experiments on real-world high-dimension noisy data (single-cell RNA-seq) to show that removing points from these datasets via our outlier detection method improves the accuracy of clustering algorithms. Our method is very competitive with popular outlier detection tools in this task.