Koopcon: A new approach towards smarter and less complex learning

TL;DR

Koopcon tackles the challenge of learning from very large image datasets by condensing data into compact, information-rich representations. It combines Autoencoder-based encoding with Koopman operator theory to linearize nonlinear data dynamics in a latent space, and uses Optimal Transport with Wasserstein distance to align the distributions of original and condensed representations. The method introduces a self-attention step, a learned linear evolution in latent space, and a multi-term loss including reconstruction, classification, OT, and covariance regularization, producing X' that yields near-parallel classifier performance to models trained on the full data. This approach enables significant computational savings and data efficiency, with demonstrated efficacy across MNIST, FashionMNIST, and CIFAR10 and potential applicability to constrained environments and large-scale systems.

Abstract

In the era of big data, the sheer volume and complexity of datasets pose significant challenges in machine learning, particularly in image processing tasks. This paper introduces an innovative Autoencoder-based Dataset Condensation Model backed by Koopman operator theory that effectively packs large datasets into compact, information-rich representations. Inspired by the predictive coding mechanisms of the human brain, our model leverages a novel approach to encode and reconstruct data, maintaining essential features and label distributions. The condensation process utilizes an autoencoder neural network architecture, coupled with Optimal Transport theory and Wasserstein distance, to minimize the distributional discrepancies between the original and synthesized datasets. We present a two-stage implementation strategy: first, condensing the large dataset into a smaller synthesized subset; second, evaluating the synthesized data by training a classifier and comparing its performance with a classifier trained on an equivalent subset of the original data. Our experimental results demonstrate that the classifiers trained on condensed data exhibit comparable performance to those trained on the original datasets, thus affirming the efficacy of our condensation model. This work not only contributes to the reduction of computational resources but also paves the way for efficient data handling in constrained environments, marking a significant step forward in data-efficient machine learning.

Figures (3)

• Figure 1: Our proposed model architecture
• Figure 2: Stages of Implementation and evaluation of a condensation model
• Figure 3: Test accuracy comparison between our model with different depth of Autoencoder, GM and DM Architectures in Different datasets

Theorems & Definitions (2)

• Theorem 1: Koopman Operator Linearity
• Theorem 2: Deep Koopman Learning