Analyzing the Structure of Handwritten Digits: A Comparative Study of PCA, Factor Analysis, and UMAP

TL;DR

This study probes the latent organization of MNIST digits in a $784$-dimensional pixel space using PCA, FA, and UMAP to understand intrinsic dimensionality and geometry. PCA identifies dominant global variance directions and enables reconstruction with a relatively small component set ($ ext{top }100$ components; first $10$ components capture about $50\%$ of variance, top $50$ exceed $80\%$). FA decomposes digits into interpretable handwriting primitives via 7 latent factors, revealing shared structural components such as loops and strokes, especially after Varimax rotation. UMAP uncovers nonlinear manifolds that reflect smooth stylistic transitions and local-to-global structure, highlighting curved and looped relationships not visible to linear methods. Together, these methods show that MNIST digits lie on a structured, low-dimensional manifold and demonstrate that linear and nonlinear approaches provide complementary perspectives useful for visualization and interpretation beyond classification.

Abstract

Handwritten digit images lie in a high-dimensional pixel space but exhibit strong geometric and statistical structure. This paper investigates the latent organization of handwritten digits in the MNIST dataset using three complementary dimensionality reduction techniques: Principal Component Analysis (PCA), Factor Analysis (FA), and Uniform Manifold Approximation and Projection (UMAP). Rather than focusing on classification accuracy, we study how each method characterizes intrinsic dimensionality, shared variation, and nonlinear geometry. PCA reveals dominant global variance directions and enables high-fidelity reconstructions using a small number of components. FA decomposes digits into interpretable latent handwriting primitives corresponding to strokes, loops, and symmetry. UMAP uncovers nonlinear manifolds that reflect smooth stylistic transitions between digit classes. Together, these results demonstrate that handwritten digits occupy a structured low-dimensional manifold and that different statistical frameworks expose complementary aspects of this structure.

Figures (12)

• Figure 1: Basic Summary Statistics
• Figure 2: Basic Pixel-Correlation
• Figure 3: Variance across PCA components.
• Figure 4: First seven PCA eigen-digits.
• Figure 5: First seven PCA eigen-digits for the digit "5".