Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Token-Level Text Anomaly Detection

Yang Cao, Bicheng Yu, Sikun Yang, Ming Liu, Yujiu Yang

TL;DR

The paper tackles the lack of fine-grained localization in text anomaly detection by formalizing token-level and document-level anomalies and proposing a unified framework. It introduces TokenCore, a memory-bank method that stores normal token embeddings and uses nearest-neighbor distances to score tokens and aggregate document-level anomalies. Three token-level annotated datasets (character corruption, semantic, and grammatical errors) accompany the method, with code released for research use. Experiments show TokenCore generally outperforms baselines on token- and document-level AUROC/AUPRC across datasets, demonstrating its potential for interpretable and precise anomaly localization in text.

Abstract

Despite significant progress in text anomaly detection for web applications such as spam filtering and fake news detection, existing methods are fundamentally limited to document-level analysis, unable to identify which specific parts of a text are anomalous. We introduce token-level anomaly detection, a novel paradigm that enables fine-grained localization of anomalies within text. We formally define text anomalies at both document and token-levels, and propose a unified detection framework that operates across multiple levels. To facilitate research in this direction, we collect and annotate three benchmark datasets spanning spam, reviews and grammar errors with token-level labels. Experimental results demonstrate that our framework get better performance than other 6 baselines, opening new possibilities for precise anomaly localization in text. All the codes and data are publicly available on https://github.com/charles-cao/TokenCore.

Towards Token-Level Text Anomaly Detection

TL;DR

The paper tackles the lack of fine-grained localization in text anomaly detection by formalizing token-level and document-level anomalies and proposing a unified framework. It introduces TokenCore, a memory-bank method that stores normal token embeddings and uses nearest-neighbor distances to score tokens and aggregate document-level anomalies. Three token-level annotated datasets (character corruption, semantic, and grammatical errors) accompany the method, with code released for research use. Experiments show TokenCore generally outperforms baselines on token- and document-level AUROC/AUPRC across datasets, demonstrating its potential for interpretable and precise anomaly localization in text.

Abstract

Despite significant progress in text anomaly detection for web applications such as spam filtering and fake news detection, existing methods are fundamentally limited to document-level analysis, unable to identify which specific parts of a text are anomalous. We introduce token-level anomaly detection, a novel paradigm that enables fine-grained localization of anomalies within text. We formally define text anomalies at both document and token-levels, and propose a unified detection framework that operates across multiple levels. To facilitate research in this direction, we collect and annotate three benchmark datasets spanning spam, reviews and grammar errors with token-level labels. Experimental results demonstrate that our framework get better performance than other 6 baselines, opening new possibilities for precise anomaly localization in text. All the codes and data are publicly available on https://github.com/charles-cao/TokenCore.
Paper Structure (10 sections, 3 equations, 1 figure, 1 table)

This paper contains 10 sections, 3 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. What are anomalies in text?
  4. Datasets
  5. Memory Bank Framework
  6. Experiment
  7. Experimental Setup
  8. Results
  9. Challenges
  10. Conclusion

Figures (1)

  • Figure 1: Demonstration of the key steps for TokenCore. Normal documents are encoded by LLM to construct a memory bank of normal token embeddings. During testing, test documents are tokenized and each token embedding is compared against the memory bank via nearest neighbor search. Token-level anomaly scores are computed based on distances to the nearest normal neighbors, and document-level scores are obtained through mean aggregation.