Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs

Fatemeh Hadadi, Qinghua Xu, Domenico Bianculli, Lionel Briand

TL;DR

FlexLog targets anomaly detection on unstable logs (ULAD) by fusing traditional ML detectors with a fine-tuned open/closed-source LLM in an ensemble framework. It introduces cache-empowered inference and retrieval-augmented prompting to adapt to log evolution while using limited labeled data. Across four ULAD datasets (ADFA-U, LOGEVOL-U, SynHDFS-U, SYNEVOL-U), FlexLog delivers state-of-the-art F1 scores with substantial data-efficiency gains (about 63–78% reduction in labeling) and maintains practical latency (under 1 second per log sequence) alongside modest memory overhead. Ablation studies show the cache, RAG, and ensemble components each contribute to performance, with Mistral emerging as a cost-effective yet competitive LLM choice. The work demonstrates a scalable, data-efficient pathway to robust log-based anomaly detection in realistic, evolving software environments, while outlining future enhancements such as more powerful open LLMs and smarter ensemble strategies.

Abstract

Most log-based anomaly detectors assume logs are stable, though logs are often unstable due to software or environmental changes. Anomaly detection on unstable logs (ULAD) is therefore a more realistic, yet under-investigated challenge. Current approaches predominantly employ machine learning (ML) models, which often require extensive labeled data for training. To mitigate data insufficiency, we propose FlexLog, a novel hybrid approach for ULAD that combines ML models -- decision tree, k-nearest neighbors, and a feedforward neural network -- with a Large Language Model (Mistral) through ensemble learning. FlexLog also incorporates a cache and retrieval-augmented generation (RAG) to further enhance efficiency and effectiveness. To evaluate FlexLog, we configured four datasets for \task, namely ADFA-U, LOGEVOL-U, SynHDFS-U, and SYNEVOL-U. FlexLog outperforms all baselines by at least 1.2 percentage points (pp) in F1 score while using much less labeled data (62.87 pp reduction). When trained on the same amount of data as the baselines, FlexLog achieves up to a 13 pp increase in F1 score on ADFA-U across varying training dataset sizes. Additionally, FlexLog maintains inference time under one second per log sequence, making it suitable for most applications, except latency-sensitive systems. Further analysis reveals the positive impact of FlexLog's key components: cache, RAG and ensemble learning.

LLM meets ML: Data-efficient Anomaly Detection on Unstable Logs

TL;DR

FlexLog targets anomaly detection on unstable logs (ULAD) by fusing traditional ML detectors with a fine-tuned open/closed-source LLM in an ensemble framework. It introduces cache-empowered inference and retrieval-augmented prompting to adapt to log evolution while using limited labeled data. Across four ULAD datasets (ADFA-U, LOGEVOL-U, SynHDFS-U, SYNEVOL-U), FlexLog delivers state-of-the-art F1 scores with substantial data-efficiency gains (about 63–78% reduction in labeling) and maintains practical latency (under 1 second per log sequence) alongside modest memory overhead. Ablation studies show the cache, RAG, and ensemble components each contribute to performance, with Mistral emerging as a cost-effective yet competitive LLM choice. The work demonstrates a scalable, data-efficient pathway to robust log-based anomaly detection in realistic, evolving software environments, while outlining future enhancements such as more powerful open LLMs and smarter ensemble strategies.

Abstract

Most log-based anomaly detectors assume logs are stable, though logs are often unstable due to software or environmental changes. Anomaly detection on unstable logs (ULAD) is therefore a more realistic, yet under-investigated challenge. Current approaches predominantly employ machine learning (ML) models, which often require extensive labeled data for training. To mitigate data insufficiency, we propose FlexLog, a novel hybrid approach for ULAD that combines ML models -- decision tree, k-nearest neighbors, and a feedforward neural network -- with a Large Language Model (Mistral) through ensemble learning. FlexLog also incorporates a cache and retrieval-augmented generation (RAG) to further enhance efficiency and effectiveness. To evaluate FlexLog, we configured four datasets for \task, namely ADFA-U, LOGEVOL-U, SynHDFS-U, and SYNEVOL-U. FlexLog outperforms all baselines by at least 1.2 percentage points (pp) in F1 score while using much less labeled data (62.87 pp reduction). When trained on the same amount of data as the baselines, FlexLog achieves up to a 13 pp increase in F1 score on ADFA-U across varying training dataset sizes. Additionally, FlexLog maintains inference time under one second per log sequence, making it suitable for most applications, except latency-sensitive systems. Further analysis reveals the positive impact of FlexLog's key components: cache, RAG and ensemble learning.
Paper Structure (58 sections, 7 equations, 7 figures, 21 tables)

This paper contains 58 sections, 7 equations, 7 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Logs
  4. Anomaly Detection on Logs
  5. Task Adaptation Strategies for Large Language Models
  6. Methodology
  7. Preprocessing
  8. Cache-empowered Inference
  9. Context-enriched Prompting
  10. Ensemble Learning
  11. Experimental Design
  12. Research Questions
  13. Experiment Setup
  14. Datasets
  15. ADFA
  16. ...and 43 more sections

Figures (7)

  • Figure 1: Examples of log Message, Log Message Sequence, Log Template, and Log Template Sequence.
  • Figure 2: Examples of Unstable Logs Resulting from Log Evolution.
  • Figure 3: Architecture of FlexLog.
  • Figure 4: FlexLog's Prompt Design for LLM Fine-tuning and Inference.
  • Figure 5: Examples of Template-level Changes
  • ...and 2 more figures

Theorems & Definitions (4)

  • Definition 2.1: Stable Logs
  • Definition 2.2: Unstable Logs
  • Definition 2.3: Anomaly Detection on Stable Logs
  • Definition 2.4: Anomaly Detection on Unstable Logs