Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Log-driven Anomaly Detection System based on Evolving Decision Making

Zhuoran Tan, Qiyuan Wang, Christos Anagnostopoulos, Shameem P. Parambath, Jeremy Singer, Sam Temple

TL;DR

CEDLog tackles scalable log anomaly detection by integrating distributed processing (Airflow and Dask) with continual learning (Elastic Weight Consolidation) and a HITL feedback loop. It employs a dual-model detector (MLP for core features and GCN for ParameterList graphs) with a fusion stage that computes $F = P(p_1 = 0)\, s_0 + P(p_2 = 0)\, s_1$ and a threshold to produce predictions, aiming to improve precision while minimizing false positives. Logs are parsed with Drain, features are engineered via Random Forest selection and semantic embeddings, and a parallel processing pipeline accelerates training and inference, all deployed in Docker for offline training and online inference. Evaluation on BGL and HDFS demonstrates high precision and low false positives, with EWC reducing catastrophic forgetting during continual updates and scalable deployment enhancing applicability across multiple clients.

Abstract

Effective anomaly detection from logs is crucial for enhancing cybersecurity defenses by enabling the early identification of threats. Despite advances in anomaly detection, existing systems often fall short in areas such as post-detection validation, scalability, and effective maintenance. These limitations not only hinder the detection of new threats but also impair overall system performance. To address these challenges, we propose CEDLog, a novel practical framework that integrates Elastic Weight Consolidation (EWC) for continual learning and implements distributed computing for scalable processing by integrating Apache Airflow and Dask. In CEDLog, anomalies are detected through the synthesis of Multi-layer Perceptron (MLP) and Graph Convolutional Networks (GCNs) using critical features present in event logs. Through comparisons with update strategies on large-scale datasets, we demonstrate the strengths of CEDLog, showcasing efficient updates and low false positives

Distributed Log-driven Anomaly Detection System based on Evolving Decision Making

TL;DR

CEDLog tackles scalable log anomaly detection by integrating distributed processing (Airflow and Dask) with continual learning (Elastic Weight Consolidation) and a HITL feedback loop. It employs a dual-model detector (MLP for core features and GCN for ParameterList graphs) with a fusion stage that computes and a threshold to produce predictions, aiming to improve precision while minimizing false positives. Logs are parsed with Drain, features are engineered via Random Forest selection and semantic embeddings, and a parallel processing pipeline accelerates training and inference, all deployed in Docker for offline training and online inference. Evaluation on BGL and HDFS demonstrates high precision and low false positives, with EWC reducing catastrophic forgetting during continual updates and scalable deployment enhancing applicability across multiple clients.

Abstract

Effective anomaly detection from logs is crucial for enhancing cybersecurity defenses by enabling the early identification of threats. Despite advances in anomaly detection, existing systems often fall short in areas such as post-detection validation, scalability, and effective maintenance. These limitations not only hinder the detection of new threats but also impair overall system performance. To address these challenges, we propose CEDLog, a novel practical framework that integrates Elastic Weight Consolidation (EWC) for continual learning and implements distributed computing for scalable processing by integrating Apache Airflow and Dask. In CEDLog, anomalies are detected through the synthesis of Multi-layer Perceptron (MLP) and Graph Convolutional Networks (GCNs) using critical features present in event logs. Through comparisons with update strategies on large-scale datasets, we demonstrate the strengths of CEDLog, showcasing efficient updates and low false positives

Paper Structure

This paper contains 18 sections, 16 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries & Problem Fundamentals
  4. Single-Point Anomaly
  5. Sequential Anomalies
  6. Model Deployment
  7. Anomaly Detection in Logs
  8. Methodology
  9. Log Parsing
  10. Feature Engineering
  11. Scalable Processing
  12. Dual-Model Detection
  13. Human-in-the-Loop Continual Learning
  14. Scalable Deployment
  15. Evaluation
  16. ...and 3 more sections

Figures (2)

  • Figure 1: The CEDLog Framework. Training Stage: a1. Read dataset with parquet format; a2.1. Analyse the features and send top weighted features to the MLP; a2.2. Create a graph based on ParameterList, train with GCN; a3. Integrate the correct predicted framework to update both models. Inference Stage: b1. Log transformation with ELK and output JSON format; b2. Log parsing and output data-frame with parquet format; b3. Choose top weighted features; b4. Separately detect with MLP for core features without Parameterlist, and GCN for Parameterlist; b5. Fuse the predicted results; b6 Analyst checks the predicted results; b7 Send the corrected result to ElasticAlert.
  • Figure 2: Airflow in Distributed Mode