Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study

TL;DR

This paper addresses the challenge that large volumes of system logs slow down and sometimes mislead anomaly detection. It empirically quantifies how log-reduction affects six supervised anomaly-detection models across three real-world datasets, identifying anti-events, duplicative-events, and key-events as distinct categories. Building on these insights, the authors introduce LogCleaner, a middleware that profiles and online filters to automatically reduce log events, achieving over 70% reduction and up to ~300% faster inference while improving detection performance across models. Experimental results across diverse datasets and models demonstrate universal benefits of log reduction and reveal the practical value of categorizing log events for robust anomaly detection in complex software systems.

Abstract

As software systems grow increasingly intricate, the precise detection of anomalies have become both essential and challenging. Current log-based anomaly detection methods depend heavily on vast amounts of log data leading to inefficient inference and potential misguidance by noise logs. However, the quantitative effects of log reduction on the effectiveness of anomaly detection remain unexplored. Therefore, we first conduct a comprehensive study on six distinct models spanning three datasets. Through the study, the impact of log quantity and their effectiveness in representing anomalies is qualifies, uncovering three distinctive log event types that differently influence model performance. Drawing from these insights, we propose LogCleaner: an efficient methodology for the automatic reduction of log events in the context of anomaly detection. Serving as middleware between software systems and models, LogCleaner continuously updates and filters anti-events and duplicative-events in the raw generated logs. Experimental outcomes highlight LogCleaner's capability to reduce over 70% of log events in anomaly detection, accelerating the model's inference speed by approximately 300%, and universally improving the performance of models for anomaly detection.

Figures (9)

• Figure 1: Log-based Anomaly Detection: The Common Workflow
• Figure 2: Process of Retry-based Approach
• Figure 3: Process of Clustering-based Approach
• Figure 4: Analysis of the Only Remaining Event (E1210) and Its Impact on Model Performance
• Figure 5: Extent of Log Event Reduction in Anomaly Detection Methods Depending on the Variation of $\alpha$