AD-LLM: Benchmarking Large Language Models for Anomaly Detection

TL;DR

This paper introduces AD-LLM, the first comprehensive benchmark for evaluating large language models in NLP anomaly detection across three core tasks: zero-shot detection, data augmentation, and model selection. Using five NLP AD datasets and multiple LLM backbones, it demonstrates that zero-shot AD can outperform some traditional methods, that carefully designed synthetic data and category descriptions can boost detector performance, and that LLMs can frequently identify strong candidate models for anomaly detection without relying on past performance data. However, it also highlights challenges in interpretability and consistency of model-selection rationales, and notes that augmentation benefits vary across detectors. The work provides a reproducible MIT-licensed framework and outlines six future research directions to advance LLM-driven anomaly detection, with implications for improving AD systems under limited supervision while balancing cost and transparency.

Abstract

Anomaly detection (AD) is an important machine learning task with many real-world uses, including fraud detection, medical diagnosis, and industrial monitoring. Within natural language processing (NLP), AD helps detect issues like spam, misinformation, and unusual user activity. Although large language models (LLMs) have had a strong impact on tasks such as text generation and summarization, their potential in AD has not been studied enough. This paper introduces AD-LLM, the first benchmark that evaluates how LLMs can help with NLP anomaly detection. We examine three key tasks: (i) zero-shot detection, using LLMs' pre-trained knowledge to perform AD without tasks-specific training; (ii) data augmentation, generating synthetic data and category descriptions to improve AD models; and (iii) model selection, using LLMs to suggest unsupervised AD models. Through experiments with different datasets, we find that LLMs can work well in zero-shot AD, that carefully designed augmentation methods are useful, and that explaining model selection for specific datasets remains challenging. Based on these results, we outline six future research directions on LLMs for AD.

Figures (3)

• Figure 1: AD-LLM examines how LLMs contribute to three key AD tasks: (Task 1, § \ref{['sec:ad']}) Zero-shot detection (left), where LLMs directly identify anomalies and provide explanations without task-specific training data; (Task 2, § \ref{['sec:aug']}) Data augmentation (center), where LLMs generate synthetic samples and produce category descriptions to alleviate data scarcity and improve semantic reasoning; and (Task 3, § \ref{['sec:ums']}) Model selection (right), where LLMs analyze dataset attributes and model descriptions to recommend suitable AD models along with justifications.
• Figure 2: Average performance over five datasets of AD baselines trained on limited data, w/ or w/o LLM-generated synthetic data, and on full datasets across five datasets. (a) Detectors that benefit from augmentation. (b) Detectors that degrade with augmentation.
• Figure 3: Model selection results across five datasets. We display the average AUROC and AUPRC of models recommended by querying each reasoning LLM five times (duplicates allowed). "Best Performance" marks the highest performance achieved by any baseline model for each dataset, while "Average Performance" denotes the mean performance across all baseline models.