Autoencoder with Ordered Variance for Nonlinear Model Identification

TL;DR

The paper tackles unsupervised nonlinear model identification by introducing an Autoencoder with Ordered Variance (AEO) that embeds a variance-regularized loss to enforce monotone latent-variance ordering via a diagonal weighting matrix $Q$, and extends this idea with ResNet-based RAEO to enable explicit nonlinear relationship extraction and avoid trivial solutions. It demonstrates that ordered latent variances can reveal nonlinear relations as implicit equations, and that RAEO can yield explicit mappings in some configurations, improving reconstruction and prediction over PCA in simulated nonlinear settings. Through 2D illustrative examples and a 5-variable simulation, the authors show that the proposed methods identify nonlinear dependencies and achieve accurate reconstructions, with ordering controlled by the parameter $q$ in $Q$. The work contributes a novel framework for simultaneous feature ordering and nonlinear model discovery in an unsupervised setting, highlighting practical implications for soft sensing, data reconciliation, and real-time optimization, while pointing to future work on parameter selection and theoretical guarantees.

Abstract

This paper presents a novel autoencoder with ordered variance (AEO) in which the loss function is modified with a variance regularization term to enforce order in the latent space. Further, the autoencoder is modified using ResNets, which results in a ResNet AEO (RAEO). The paper also illustrates the effectiveness of AEO and RAEO in extracting nonlinear relationships among input variables in an unsupervised setting.

Figures (8)

• Figure 1: (a) PCA block diagram (b) PCA illustration.
• Figure 2: (a) AE block diagram (b) AE illustration.
• Figure 3: Response of 2-variable example with AEO.
• Figure 4: (a) ResNet (b) Weighted ResNet.
• Figure 5: ResNet Autoencoder.

Theorems & Definitions (3)

• Remark 1
• Remark 2
• Remark 3