Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection

Jing-Xiao Liao, Bo-Jian Hou, Hang-Cheng Dong, Hao Zhang, Xiaoge Zhang, Jinwei Sun, Shiping Zhang, Feng-Lei Fan

TL;DR

This work addresses unsupervised anomaly detection in high-dimensional data by introducing neuronal diversity through a heterogeneous autoencoder (HAE) that combines conventional and quadratic neurons. The authors provide a theoretical foundation showing a polynomial-width heterogeneous network can efficiently approximate certain target functions that are hard for purely conventional or purely quadratic networks, motivating the architecture. They propose three symmetric HAE designs and employ a ReLinear training scheme to stabilize the learning of quadratic components. Empirically, HAEs achieve competitive or superior performance compared with state-of-the-art baselines on tabular anomaly datasets and bearing fault data, and ablation studies confirm the advantage of heterogeneity and first-layer quadratic neurons. Overall, the paper demonstrates that mixing neuron types can enhance representation and anomaly-detection capabilities in unsupervised settings, with potential broad impact on real-world monitoring tasks.

Abstract

Inspired by the complexity and diversity of biological neurons, a quadratic neuron is proposed to replace the inner product in the current neuron with a simplified quadratic function. Employing such a novel type of neurons offers a new perspective on developing deep learning. When analyzing quadratic neurons, we find that there exists a function such that a heterogeneous network can approximate it well with a polynomial number of neurons but a purely conventional or quadratic network needs an exponential number of neurons to achieve the same level of error. Encouraged by this inspiring theoretical result on heterogeneous networks, we directly integrate conventional and quadratic neurons in an autoencoder to make a new type of heterogeneous autoencoders. To our best knowledge, it is the first heterogeneous autoencoder that is made of different types of neurons. Next, we apply the proposed heterogeneous autoencoder to unsupervised anomaly detection for tabular data and bearing fault signals. The anomaly detection faces difficulties such as data unknownness, anomaly feature heterogeneity, and feature unnoticeability, which is suitable for the proposed heterogeneous autoencoder. Its high feature representation ability can characterize a variety of anomaly data (heterogeneity), discriminate the anomaly from the normal (unnoticeability), and accurately learn the distribution of normal samples (unknownness). Experiments show that heterogeneous autoencoders perform competitively compared to other state-of-the-art models.

Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection

TL;DR

This work addresses unsupervised anomaly detection in high-dimensional data by introducing neuronal diversity through a heterogeneous autoencoder (HAE) that combines conventional and quadratic neurons. The authors provide a theoretical foundation showing a polynomial-width heterogeneous network can efficiently approximate certain target functions that are hard for purely conventional or purely quadratic networks, motivating the architecture. They propose three symmetric HAE designs and employ a ReLinear training scheme to stabilize the learning of quadratic components. Empirically, HAEs achieve competitive or superior performance compared with state-of-the-art baselines on tabular anomaly datasets and bearing fault data, and ablation studies confirm the advantage of heterogeneity and first-layer quadratic neurons. Overall, the paper demonstrates that mixing neuron types can enhance representation and anomaly-detection capabilities in unsupervised settings, with potential broad impact on real-world monitoring tasks.

Abstract

Inspired by the complexity and diversity of biological neurons, a quadratic neuron is proposed to replace the inner product in the current neuron with a simplified quadratic function. Employing such a novel type of neurons offers a new perspective on developing deep learning. When analyzing quadratic neurons, we find that there exists a function such that a heterogeneous network can approximate it well with a polynomial number of neurons but a purely conventional or quadratic network needs an exponential number of neurons to achieve the same level of error. Encouraged by this inspiring theoretical result on heterogeneous networks, we directly integrate conventional and quadratic neurons in an autoencoder to make a new type of heterogeneous autoencoders. To our best knowledge, it is the first heterogeneous autoencoder that is made of different types of neurons. Next, we apply the proposed heterogeneous autoencoder to unsupervised anomaly detection for tabular data and bearing fault signals. The anomaly detection faces difficulties such as data unknownness, anomaly feature heterogeneity, and feature unnoticeability, which is suitable for the proposed heterogeneous autoencoder. Its high feature representation ability can characterize a variety of anomaly data (heterogeneity), discriminate the anomaly from the normal (unnoticeability), and accurately learn the distribution of normal samples (unknownness). Experiments show that heterogeneous autoencoders perform competitively compared to other state-of-the-art models.
Paper Structure (8 sections, 5 theorems, 14 equations, 6 figures, 15 tables)

This paper contains 8 sections, 5 theorems, 14 equations, 6 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Power of Heterogeneous Networks
  4. Heterogeneous Autoencoder
  5. Experiments
  6. Ablation Study
  7. Analysis Experiments
  8. Conclusions

Key Result

Theorem 1

There exist universal constants $c_\sigma$, $c_1,c_2,c_3,c_4, C_1, C_2, \epsilon_1, \epsilon_2>0$ such that the following holds: For every dimension $d$, there exists a measure $\mu$ and a function $\tilde{g}_{ip}(\boldsymbol{x})+ \tilde{g}_{r}(\boldsymbol{x}): \mathbb{R}^d \to \mathbb{R}$ with the

Figures (6)

  • Figure 1: Fourier transforms of a radial function $f$ and an inner-product based function $g$ in the frequency domain.
  • Figure 2: Three heterogeneous autoencoder designs.
  • Figure 3: ReLinear: a training strategy of quadratic networks (R. I. means random initialization).
  • Figure 4: The measurement structure of seawater booster pump.
  • Figure 5: Abnormal and normal signals of the seawater booster pump.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Theorem 1: Main
  • Theorem 2
  • Theorem 3: Theorem 1 of fan2020universal
  • proof
  • Lemma 1
  • proof
  • Lemma 2
  • proof