Explicit and data-Efficient Encoding via Gradient Flow

TL;DR

Encoder inversion in traditional autoencoders can yield suboptimal latent representations, especially in data-scarce physical sciences. This work proposes gradient flow encoding (GFE), a decoder-only framework where each input's latent representation $z^*$ is obtained by integrating a gradient-flow ODE up to time $\tau$, and the decoder is trained via the adjoint method using $\mathcal{L}(\theta) = \sum_m l(y_m, D(z_m^*, \theta))$. It introduces a $2^{nd}$ order ODE variant to approximate Nesterov’s accelerated gradient flow and an Adaptive minimise distance (AMD) solver to cope with stiffness while prioritizing loss reduction. Empirically, GFE-amd delivers superior data efficiency—reconstructing with far fewer examples than a standard autoencoder and remaining competitive on multiple datasets when fully trained—while also reducing network size by removing the encoder. The approach holds promise for integrating machine learning into scientific workflows requiring precise, data-efficient encoding; code is available at the provided repository.

Abstract

The autoencoder model typically uses an encoder to map data to a lower dimensional latent space and a decoder to reconstruct it. However, relying on an encoder for inversion can lead to suboptimal representations, particularly limiting in physical sciences where precision is key. We introduce a decoder-only method using gradient flow to directly encode data into the latent space, defined by ordinary differential equations (ODEs). This approach eliminates the need for approximate encoder inversion. We train the decoder via the adjoint method and show that costly integrals can be avoided with minimal accuracy loss. Additionally, we propose a $2^{nd}$ order ODE variant, approximating Nesterov's accelerated gradient descent for faster convergence. To handle stiff ODEs, we use an adaptive solver that prioritizes loss minimization, improving robustness. Compared to traditional autoencoders, our method demonstrates explicit encoding and superior data efficiency, which is crucial for data-scarce scenarios in the physical sciences. Furthermore, this work paves the way for integrating machine learning into scientific workflows, where precise and efficient encoding is critical. \footnote{The code for this work is available at \url{https://github.com/k-flouris/gfe}.}

Figures (3)

• Figure 1: Left Validation mean cross-entropy loss vs. number of MNIST training images for GFE-amd and AE methods, with GFE-amd showing significant convergence with minimal training data. Right Validation mean cross-entropy loss vs. time for GFE-amd and AE methods, with AE being faster due to more iterations in the same time span.
• Figure 2: Left Validation mean cross-entropy loss plotted against MNIST training iterations for the approximate and full adjoint GFE methods. The full adjoint has a slight advantage over the approximate. Right Validation mean cross-entropy loss plotted against MNIST training iterations for the GFE, $2^{nd}$ order GFE and GFE-amd methods. The GFE-amd is both more stable and approaches a better convergence relative to the other methods
• Figure 3: (a) Test-set reconstructions for trained GFE-amd (left) and AE (right) that only see $1\%$ of MNIST (top) and FashionMNIST (bottom) training images. (b) Test-set reconstructions for fully trained GFE-amd (left) and AE (right) with MNIST (top) and FashionMNIST (bottom) training images. Note: The labels are identical in the respective reconstructions.