Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study

TL;DR

This work tackles spectral bias in autoencoders and variational autoencoders by incorporating Random Fourier Transformations (RFT) and a trainable Fourier variant (TFT). It demonstrates that RFT/TFT enable simultaneous learning of low- and high-frequency components, accelerating convergence and improving reconstruction-based anomaly detection, particularly on a high-dimensional aviation safety dataset (Dashlink). While Fourier methods generally outperform vanilla networks, training the Fourier parameters (TFT) yields modest gains over the random variant, indicating limited benefits of gradient-based tuning in this setting. The findings suggest Fourier-domain augmentations meaningfully enhance representation learning and anomaly detection, with practical implications for aviation safety analytics and similar high-dimensional time-series domains.

Abstract

In this study, we focus on the training process and inference improvements of deep neural networks (DNNs), specifically Autoencoders (AEs) and Variational Autoencoders (VAEs), using Random Fourier Transformation (RFT). We further explore the role of RFT in model training behavior using Frequency Principle (F-Principle) analysis and show that models with RFT turn to learn low frequency and high frequency at the same time, whereas conventional DNNs start from low frequency and gradually learn (if successful) high-frequency features. We focus on reconstruction-based anomaly detection using autoencoder and variational autoencoder and investigate the RFT's role. We also introduced a trainable variant of RFT that uses the existing computation graph to train the expansion of RFT instead of it being random. We showcase our findings with two low-dimensional synthetic datasets for data representation, and an aviation safety dataset, called Dashlink, for high-dimensional reconstruction-based anomaly detection. The results indicate the superiority of models with Fourier transformation compared to the conventional counterpart and remain inconclusive regarding the benefits of using trainable Fourier transformation in contrast to the Random variant.

Figures (11)

• Figure 1: Schematic architecture of autoencoders. Given the input data ($x$), latent representation ($z$) and reconstructed input data ($\hat{x}$), the $g_{\phi}(.)$, $f_{\theta}(.)$ represent encoder and decoder components, respectively.
• Figure 2: Schematic architecture of autoencoders. Given the input data ($x$), latent representation ($z$) and reconstructed input data ($\hat{x}$), the $g_{\phi}(.)$, $f_{\theta}(.)$ represent encoder and decoder components, respectively.
• Figure 3: Comparison of model fitting performance using vanilla neural networks, neural networks with Trainable Fourier transformations, and neural networks with Trainable Fourier Transformations on both Step and mixture of Sines datasets (from top row to bottom). Plots depict the convergence of models (red line) to ground truth (blue line) across increasing training epochs, from left to right
• Figure 4: With a similar visualization structure to Fig. \ref{['fig:signals']}, frequency analysis results showcase the learning behavior of neural networks with and without Fourier transformations. The figure illustrates that while the vanilla neural network learns low-pass frequencies before high-pass frequencies, the neural networks with Random Fourier Transformation and Trainable Fourier Transformation exhibit simultaneous learning of both low and high frequencies.
• Figure 5: The frequency behavior of CAE, CAE with RFT, and CAE with TFT (first row) for the nominal of the Flaps anomaly dataset. Similar plots follow from the third row for CVAE, CVAE with RFT, and CVAE with TFT (second row). The y-axis in each plot represents the separation of low and high frequencies and the x-axis shows the timesteps of training evolution.