Evaluating Large Language Models for Time Series Anomaly Detection in Aerospace Software

TL;DR

This work tackles the challenge of evaluating large language models for time series anomaly detection in aerospace by introducing ATSADBench, a nine-task, 108,000-point benchmark that covers univariate and multivariate telemetry in in-loop and out-of-loop settings, under Direct and Prediction-Based LLM paradigms. It also proposes three window-based metrics—Alarm Accuracy, Alarm Latency, and Alarm Contiguity—to reflect operational alarm requirements and examines enhancement strategies such as few-shot prompting and retrieval-augmented generation. Through comprehensive experiments with open-source LLMs and state-of-the-art unsupervised baselines, the study finds that LLMs excel on univariate tasks but struggle with complex multivariate telemetry, with AA and AC near random on multivariate tasks, while few-shot prompting provides modest gains and RAG yields limited improvements. A real-world in-orbit anomaly case study demonstrates that LLMs can provide timely anomaly alarms, though false alarms can occur, and domain-informed prompting can mitigate false positives, collectively offering practical guidance for deploying LLM-based TSAD in aerospace contexts.

Abstract

Time series anomaly detection (TSAD) is essential for ensuring the safety and reliability of aerospace software systems. Although large language models (LLMs) provide a promising training-free alternative to unsupervised approaches, their effectiveness in aerospace settings remains under-examined because of complex telemetry, misaligned evaluation metrics, and the absence of domain knowledge. To address this gap, we introduce ATSADBench, the first benchmark for aerospace TSAD. ATSADBench comprises nine tasks that combine three pattern-wise anomaly types, univariate and multivariate signals, and both in-loop and out-of-loop feedback scenarios, yielding 108,000 data points. Using this benchmark, we systematically evaluate state-of-the-art open-source LLMs under two paradigms: Direct, which labels anomalies within sliding windows, and Prediction-Based, which detects anomalies from prediction errors. To reflect operational needs, we reformulate evaluation at the window level and propose three user-oriented metrics: Alarm Accuracy (AA), Alarm Latency (AL), and Alarm Contiguity (AC), which quantify alarm correctness, timeliness, and credibility. We further examine two enhancement strategies, few-shot learning and retrieval-augmented generation (RAG), to inject domain knowledge. The evaluation results show that (1) LLMs perform well on univariate tasks but struggle with multivariate telemetry, (2) their AA and AC on multivariate tasks approach random guessing, (3) few-shot learning provides modest gains whereas RAG offers no significant improvement, and (4) in practice LLMs can detect true anomaly onsets yet sometimes raise false alarms, which few-shot prompting mitigates but RAG exacerbates. These findings offer guidance for future LLM-based TSAD in aerospace software.

Figures (7)

• Figure 1: A missed detection of a fixed-value anomaly in star sensor attitude measurements can propagate through the control loop, causing erroneous commands and continuous thruster firing, ultimately resulting in critical satellite damage.
• Figure 2: Pipeline of time series anomaly detection with zero-shot LLMs
• Figure 3: Overview of time series anomaly types.
• Figure 4: Visualization of three types of pattern-wise anomaly.
• Figure 5: Conceptual illustration of the AL and AC metrics.