Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Enhanced Reinforcement Learning for Time Series Anomaly Detection

Bahareh Golchin, Banafsheh Rekabdar, Danielle Justo

TL;DR

This paper tackles time series anomaly detection under limited labels by unifying LSTM-based reinforcement learning with an LLM-based semantic potential function, VAE-driven dynamic reward scaling, and active learning with label propagation. The RL agent uses a combined reward $R_{\text{total}}(s_t,a_t) = R_1(s_t,a_t) + \lambda(t) R_2(s_t)$ and PBRS via $r'(s_t,a_t)= r(s_t,a_t) + \gamma \phi(s_{t+1}) - \phi(s_t)$ to guide exploration toward anomalous patterns. Empirical results on Yahoo-A1 and SMD show that Llama-3-based shaping delivers favorable precision-recall trade-offs, achieving state-of-the-art performance under constrained labeling budgets. This framework offers a scalable, data-efficient approach for real-world anomaly detection where labeled data are scarce and timely decisions are critical.

Abstract

Detecting anomalies in time series data is crucial for finance, healthcare, sensor networks, and industrial monitoring applications. However, time series anomaly detection often suffers from sparse labels, complex temporal patterns, and costly expert annotation. We propose a unified framework that integrates Large Language Model (LLM)-based potential functions for reward shaping with Reinforcement Learning (RL), Variational Autoencoder (VAE)-enhanced dynamic reward scaling, and active learning with label propagation. An LSTM-based RL agent leverages LLM-derived semantic rewards to guide exploration, while VAE reconstruction errors add unsupervised anomaly signals. Active learning selects the most uncertain samples, and label propagation efficiently expands labeled data. Evaluations on Yahoo-A1 and SMD benchmarks demonstrate that our method achieves state-of-the-art detection accuracy under limited labeling budgets and operates effectively in data-constrained settings. This study highlights the promise of combining LLMs with RL and advanced unsupervised techniques for robust, scalable anomaly detection in real-world applications.

LLM-Enhanced Reinforcement Learning for Time Series Anomaly Detection

TL;DR

This paper tackles time series anomaly detection under limited labels by unifying LSTM-based reinforcement learning with an LLM-based semantic potential function, VAE-driven dynamic reward scaling, and active learning with label propagation. The RL agent uses a combined reward and PBRS via to guide exploration toward anomalous patterns. Empirical results on Yahoo-A1 and SMD show that Llama-3-based shaping delivers favorable precision-recall trade-offs, achieving state-of-the-art performance under constrained labeling budgets. This framework offers a scalable, data-efficient approach for real-world anomaly detection where labeled data are scarce and timely decisions are critical.

Abstract

Detecting anomalies in time series data is crucial for finance, healthcare, sensor networks, and industrial monitoring applications. However, time series anomaly detection often suffers from sparse labels, complex temporal patterns, and costly expert annotation. We propose a unified framework that integrates Large Language Model (LLM)-based potential functions for reward shaping with Reinforcement Learning (RL), Variational Autoencoder (VAE)-enhanced dynamic reward scaling, and active learning with label propagation. An LSTM-based RL agent leverages LLM-derived semantic rewards to guide exploration, while VAE reconstruction errors add unsupervised anomaly signals. Active learning selects the most uncertain samples, and label propagation efficiently expands labeled data. Evaluations on Yahoo-A1 and SMD benchmarks demonstrate that our method achieves state-of-the-art detection accuracy under limited labeling budgets and operates effectively in data-constrained settings. This study highlights the promise of combining LLMs with RL and advanced unsupervised techniques for robust, scalable anomaly detection in real-world applications.
Paper Structure (14 sections, 5 equations, 2 figures, 2 tables)

This paper contains 14 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Extrinsic and Intrinsic Rewards
  4. Potential-Based Reward Shaping
  5. Variational Autoencoder for Anomaly Detection
  6. Proposed Method
  7. LSTM-based RL agent
  8. VAE-Enhanced Reward Augmentation
  9. LLM-Based Potential Function
  10. Active Learning Integration
  11. Experiments
  12. Datasets
  13. Results and Discussions
  14. Conclusion

Figures (2)

  • Figure 1: Our proposed framework: an LSTM-based RL agent operates on sliding windows of the time series; the reward merges a VAE-based reconstruction term and an LLM potential for semantic shaping with labels supplied via active learning, producing anomaly predictions.
  • Figure 2: SMD anomaly detection — Llama. Colors: blue = original signal; green = true (ground-truth) anomalies; red = detected anomalies. Panels: top-left shows ground truth over the signal; top-right shows ground truth + detected anomalies; bottom-left and bottom-right are two zoomed-in regions.