HELP: Hierarchical Embeddings-based Log Parsing
Andy Xu, Arno Gau
TL;DR
This work tackles the challenge of real-time log parsing in large, dynamic software systems by introducing HELP, a Hierarchical Embeddings-based Log Parser. HELP combines a fine-tuned embedding module (augmented OpenAI embeddings with a two-layer encoder and word-count feature) with an online clustering strategy, a context-aware parsing module that leverages few-shot prompting and CoT reasoning, and an iterative rebalancing component to mitigate template drift, significantly reducing LLM query costs by clustering logs first. Across 14 large Loghub-2.0 datasets, HELP outperforms state-of-the-art online parsers in grouping and template accuracy, while remaining competitive with fine-tuned offline models; it also achieves real-time performance with a P95 latency of about $1.5$ seconds and has been deployed in Iudex’s production observability platform. These results demonstrate the practical viability of online semantic-based log parsing at high throughput, offering substantial benefits for anomaly detection, troubleshooting, and root-cause analysis in real-world systems.
Abstract
Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing, which converts semi-structured log messages into structured templates, is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. However, existing log parsers fail in real-world systems for three main reasons. First, traditional heuristics-based parsers require handcrafted features and domain knowledge, which are difficult to generalize at scale. Second, existing large language model-based parsers rely on periodic offline processing, limiting their effectiveness in real-time use cases. Third, existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies. To address these challenges, we propose HELP, a Hierarchical Embeddings-based Log Parser. HELP is the first online semantic-based parser to leverage LLMs for performant and cost-effective log parsing. We achieve this through a novel hierarchical embeddings module, which fine-tunes a text embedding model to cluster logs before parsing, reducing querying costs by multiple orders of magnitude. To combat log drift, we also develop an iterative rebalancing module, which periodically updates existing log groupings. We evaluate HELP extensively on 14 public large-scale datasets, showing that HELP achieves significantly higher F1-weighted grouping and parsing accuracy than current state-of-the-art online log parsers. We also implement HELP into Iudex's production observability platform, confirming HELP's practicality in a production environment. Our results show that HELP is effective and efficient for high-throughput real-world log parsing.