MADLLM: Multivariate Anomaly Detection via Pre-trained LLMs

TL;DR

MADLLM tackles multivariate time-series anomaly detection by aligning MTS data with the text modality of pre-trained LLMs using a triple encoding scheme. It combines traditional patch embedding with two novel embeddings—Skip Embedding and Feature Embedding—enabling the model to capture cross-feature correlations and long-range history, with token sequences of length proportional to $P \times M$ where $P$ is the number of patches per feature and $M$ features. A contrastive learning encoder with a patch-based triplet loss and an exponentially causal CNN backbone learns feature embeddings, which are fused with LLM inputs while freezing the LLM body for efficient fine-tuning. Experimental results on five public datasets show that MADLLM achieves state-of-the-art AUC and competitive F1, with strong few-shot performance and substantially reduced training time.

Abstract

When applying pre-trained large language models (LLMs) to address anomaly detection tasks, the multivariate time series (MTS) modality of anomaly detection does not align with the text modality of LLMs. Existing methods simply transform the MTS data into multiple univariate time series sequences, which can cause many problems. This paper introduces MADLLM, a novel multivariate anomaly detection method via pre-trained LLMs. We design a new triple encoding technique to align the MTS modality with the text modality of LLMs. Specifically, this technique integrates the traditional patch embedding method with two novel embedding approaches: Skip Embedding, which alters the order of patch processing in traditional methods to help LLMs retain knowledge of previous features, and Feature Embedding, which leverages contrastive learning to allow the model to better understand the correlations between different features. Experimental results demonstrate that our method outperforms state-of-the-art methods in various public anomaly detection datasets.

Figures (9)

• Figure 1: An example of anomalies in the SMD dataset. The four features are marked as Feature1-4 due to dataset privacy.
• Figure 2: The architecture of MADLLM.
• Figure 3: The workflow of skip embedding and traditional patch embedding.
• Figure 4: The workflow of feature embedding.
• Figure 5: Ablation study results on different datasets.