Principal Orthogonal Latent Components Analysis (POLCA Net)

TL;DR

POLCA Net combines an autoencoder framework with a set of specialized loss functions to achieve effective dimensionality reduction, orthogonality, variance-based feature sorting, high-fidelity reconstructions, and additionally, a latent representation well suited for linear classifiers and low dimensional visualization of class distribution as well.

Abstract

Representation learning is a pivotal area in the field of machine learning, focusing on the development of methods to automatically discover the representations or features needed for a given task from raw data. Unlike traditional feature engineering, which requires manual crafting of features, representation learning aims to learn features that are more useful and relevant for tasks such as classification, prediction, and clustering. We introduce Principal Orthogonal Latent Components Analysis Network (POLCA Net), an approach to mimic and extend PCA and LDA capabilities to non-linear domains. POLCA Net combines an autoencoder framework with a set of specialized loss functions to achieve effective dimensionality reduction, orthogonality, variance-based feature sorting, high-fidelity reconstructions, and additionally, when used with classification labels, a latent representation well suited for linear classifiers and low dimensional visualization of class distribution as well.

Figures (3)

• Figure 1: A general autoencoder setting with POLCA Net as the central (bottleneck) component.
• Figure 2: Comparison of PCA and POLCA performance across all datasets. (a) Distribution of classification accuracy for PCA and POLCA for each classifier: Perceptron, Ridge Classifier, Logistic Regression, and Linear SVM. (b) Distribution of image reconstruction metrics (NRMSE, PSNR, SSIM) for PCA and POLCA, showing their relative performance in compression and reconstruction.
• Figure 3: Pairwise analysis of POLCA multiobjective loss statistics: reconstruction loss ($L_{rec}$), orthogonality loss ($L_{ort}$), dimensionality reduction center of mass loss ($L_{com}$) and variance regularization loss ($L_{var}$), collected during training phase on all the experiments realized. The gradient similarity $(s)$ and conflicts are defined in Equation \ref{['eq:loss_similairty']}

Theorems & Definitions (3)

• Definition 1: Linear Independence of Functions, axler2015linear
• Definition 2: Orthogonality of Functions, debnath2005introduction
• Definition 3: Functional Independence, hirsch1976differentiallee2012introduction