Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection

Yang Cao, Sikun Yang, Chen Li, Haolong Xiang, Lianyong Qi, Bo Liu, Rongsheng Li, Ming Liu

TL;DR

We address the challenge of evaluating embedding-based text anomaly detection by introducing TAD-Bench, a multi-domain benchmark that combines diverse datasets with a suite of embedding models and anomaly detectors. The study systematically analyzes interactions between embeddings and detectors, revealing that OpenAI embeddings offer robust, generalizable performance across tasks, while detectors like kNN and INNE provide strong reliability in high-dimensional spaces. Results highlight domain-specific differences: spam detection benefits from explicit pattern cues, whereas hate speech and contextual misinformation remain difficult. These findings inform design choices for robust, scalable NLP anomaly detection and motivate future work on adaptive embeddings and hybrid detection approaches.

Abstract

Text anomaly detection is crucial for identifying spam, misinformation, and offensive language in natural language processing tasks. Despite the growing adoption of embedding-based methods, their effectiveness and generalizability across diverse application scenarios remain under-explored. To address this, we present TAD-Bench, a comprehensive benchmark designed to systematically evaluate embedding-based approaches for text anomaly detection. TAD-Bench integrates multiple datasets spanning different domains, combining state-of-the-art embeddings from large language models with a variety of anomaly detection algorithms. Through extensive experiments, we analyze the interplay between embeddings and detection methods, uncovering their strengths, weaknesses, and applicability to different tasks. These findings offer new perspectives on building more robust, efficient, and generalizable anomaly detection systems for real-world applications.

TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection

TL;DR

We address the challenge of evaluating embedding-based text anomaly detection by introducing TAD-Bench, a multi-domain benchmark that combines diverse datasets with a suite of embedding models and anomaly detectors. The study systematically analyzes interactions between embeddings and detectors, revealing that OpenAI embeddings offer robust, generalizable performance across tasks, while detectors like kNN and INNE provide strong reliability in high-dimensional spaces. Results highlight domain-specific differences: spam detection benefits from explicit pattern cues, whereas hate speech and contextual misinformation remain difficult. These findings inform design choices for robust, scalable NLP anomaly detection and motivate future work on adaptive embeddings and hybrid detection approaches.

Abstract

Text anomaly detection is crucial for identifying spam, misinformation, and offensive language in natural language processing tasks. Despite the growing adoption of embedding-based methods, their effectiveness and generalizability across diverse application scenarios remain under-explored. To address this, we present TAD-Bench, a comprehensive benchmark designed to systematically evaluate embedding-based approaches for text anomaly detection. TAD-Bench integrates multiple datasets spanning different domains, combining state-of-the-art embeddings from large language models with a variety of anomaly detection algorithms. Through extensive experiments, we analyze the interplay between embeddings and detection methods, uncovering their strengths, weaknesses, and applicability to different tasks. These findings offer new perspectives on building more robust, efficient, and generalizable anomaly detection systems for real-world applications.
Paper Structure (22 sections, 2 equations, 6 figures, 6 tables)

This paper contains 22 sections, 2 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text representations
  4. Anomaly Detection
  5. Benchmark Settings
  6. Datasets
  7. Embedding Models
  8. Anomaly Detectors
  9. Evaluation Criteria and Trials
  10. Experiments
  11. Domain Generalization
  12. Comparative Effectiveness of Embeddings in Anomaly Detection
  13. Performance Across Anomaly Detectors
  14. Conclusion
  15. Problem Definitions
  16. ...and 7 more sections

Figures (6)

  • Figure 1: t-SNE visualization of SMS_Spam dataset's embedding extracted by OpenAI model. Blue and red points are normal and anomaly points, respectively.
  • Figure 2: Illustration of the embedding-based anomaly detection pipeline, encompassing embedding extraction and anomaly scoring.
  • Figure 3: t-SNE demonstration of Case 1 (green star) embedding extracted by O-large.
  • Figure 4: t-SNE demonstration of Case 2 (green star) embedding extracted by O-large.
  • Figure 5: Boxplot of AUCROC scores for anomaly detectors on different embeddings across 6 datasets.
  • ...and 1 more figures