Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text Anomaly Detection with Simplified Isolation Kernel

Yang Cao, Sikun Yang, Yujiu Yang, Lianyong Qi, Ming Liu

TL;DR

This work tackles the challenge of text anomaly detection when leveraging high-dimensional embeddings from large language models. It introduces the Simplified Isolation Kernel (SIK), a boundary-focused, low-dimensional sparse representation that preserves crucial anomaly characteristics while achieving linear time and space complexity. The authors prove SIK is a valid kernel and demonstrate through extensive experiments on seven datasets that SIK often outperforms 11 SOTA methods, especially when paired with OpenAI embeddings, while offering substantial scalability benefits over prior IK/IDK approaches. The results imply strong practical impact for fast, memory-efficient anomaly detection in large-scale text corpora, with publicly available code for reproducibility.

Abstract

Two-step approaches combining pre-trained large language model embeddings and anomaly detectors demonstrate strong performance in text anomaly detection by leveraging rich semantic representations. However, high-dimensional dense embeddings extracted by large language models pose challenges due to substantial memory requirements and high computation time. To address this challenge, we introduce the Simplified Isolation Kernel (SIK), which maps high-dimensional dense embeddings to lower-dimensional sparse representations while preserving crucial anomaly characteristics. SIK has linear time complexity and significantly reduces space complexity through its innovative boundary-focused feature mapping. Experiments across 7 datasets demonstrate that SIK achieves better detection performance than 11 state-of-the-art (SOTA) anomaly detection algorithms while maintaining computational efficiency and low memory cost. All code and demonstrations are available at https://github.com/charles-cao/SIK.

Text Anomaly Detection with Simplified Isolation Kernel

TL;DR

This work tackles the challenge of text anomaly detection when leveraging high-dimensional embeddings from large language models. It introduces the Simplified Isolation Kernel (SIK), a boundary-focused, low-dimensional sparse representation that preserves crucial anomaly characteristics while achieving linear time and space complexity. The authors prove SIK is a valid kernel and demonstrate through extensive experiments on seven datasets that SIK often outperforms 11 SOTA methods, especially when paired with OpenAI embeddings, while offering substantial scalability benefits over prior IK/IDK approaches. The results imply strong practical impact for fast, memory-efficient anomaly detection in large-scale text corpora, with publicly available code for reproducibility.

Abstract

Two-step approaches combining pre-trained large language model embeddings and anomaly detectors demonstrate strong performance in text anomaly detection by leveraging rich semantic representations. However, high-dimensional dense embeddings extracted by large language models pose challenges due to substantial memory requirements and high computation time. To address this challenge, we introduce the Simplified Isolation Kernel (SIK), which maps high-dimensional dense embeddings to lower-dimensional sparse representations while preserving crucial anomaly characteristics. SIK has linear time complexity and significantly reduces space complexity through its innovative boundary-focused feature mapping. Experiments across 7 datasets demonstrate that SIK achieves better detection performance than 11 state-of-the-art (SOTA) anomaly detection algorithms while maintaining computational efficiency and low memory cost. All code and demonstrations are available at https://github.com/charles-cao/SIK.
Paper Structure (22 sections, 11 equations, 3 figures, 6 tables)

This paper contains 22 sections, 11 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Representations
  4. End-to-end TAD Approaches
  5. Two-step Approaches
  6. Preliminaries
  7. Problem Definition
  8. Isolation Kernel (IK)
  9. Isolation Distributioal Kernel (IDK)
  10. Methodology
  11. Space Partitioning
  12. Feature Map
  13. Anomaly Scores Calculation
  14. Is SIK a Valid Kernel?
  15. Relationship to SiNNE
  16. ...and 7 more sections

Figures (3)

  • Figure 1: An illustration of feature maps of IK and SIK with one partitioning ($t=1$) of 3 hyperspheres. Each center of a hypersphere is at a point $z \in D$ where $\psi = 3$ are randomly selected from the given dataset $D$. When a point $x$ falls into an overlapping region, $x$ is regarded as being in the hypersphere whose center is closer to $x$. On the left IK feature space, $x$ has a 3-dimensional feature vector. On the right SIK feature space, $x$ has only a 1-dimensional feature vector.
  • Figure 2: Friedman-Nemenyi test for the anomaly detection methods based on BERT, OpenAI embeddings and end-to-end at significance level 0.1 (the lower the better).
  • Figure 3: The performance of 5 runs of SIK on Email_Spam OpenAI embedding. The anomaly ratio is the ratio of the number of normal points and anomalies in the given dataset.

Theorems & Definitions (6)

  • Definition 1
  • Definition 2: Feature Map of IK
  • Definition 3
  • Definition 4: Hypersphere Partionings
  • Definition 5: Feature Map of SIK
  • Definition 6: Anomaly Score