HLogformer: A Hierarchical Transformer for Representing Log Data

TL;DR

HLogformer addresses the difficulty of applying transformers to log data by introducing a dynamic hierarchical transformer that respects the nested, dictionary-like structure of logs. It processes logs along their hierarchical tree, using bidirectional summary passing and a memory-efficient compression scheme to achieve scalable representations. The training combines self-supervised masked language modeling with volume hypersphere minimization, enabling effective anomaly detection and downstream tasks such as supervised classification and product recommendation. The approach demonstrates memory efficiency and improved hierarchical contextual encoding across diverse datasets like CloudTrail, TrailDiscover, and Amazon Reviews, offering practical impact for enterprise log analytics.

Abstract

Transformers have gained widespread acclaim for their versatility in handling diverse data structures, yet their application to log data remains underexplored. Log data, characterized by its hierarchical, dictionary-like structure, poses unique challenges when processed using conventional transformer models. Traditional methods often rely on manually crafted templates for parsing logs, a process that is labor-intensive and lacks generalizability. Additionally, the linear treatment of log sequences by standard transformers neglects the rich, nested relationships within log entries, leading to suboptimal representations and excessive memory usage. To address these issues, we introduce HLogformer, a novel hierarchical transformer framework specifically designed for log data. HLogformer leverages the hierarchical structure of log entries to significantly reduce memory costs and enhance representation learning. Unlike traditional models that treat log data as flat sequences, our framework processes log entries in a manner that respects their inherent hierarchical organization. This approach ensures comprehensive encoding of both fine-grained details and broader contextual relationships. Our contributions are threefold: First, HLogformer is the first framework to design a dynamic hierarchical transformer tailored for dictionary-like log data. Second, it dramatically reduces memory costs associated with processing extensive log sequences. Third, comprehensive experiments demonstrate that HLogformer more effectively encodes hierarchical contextual information, proving to be highly effective for downstream tasks such as synthetic anomaly detection and product recommendation.

Figures (7)

• Figure 1: Different representations of log data: (a) treating log data as a sequence, and (b) treating the log data as a hierarchical tree.
• Figure 2: Schematic overview of HLogformer: HLogformer encapsulates the context segment into a summary vector, which is then passed from low-level to high-level (left). Specifically, at each step, we concatenate all the child nodes' tokens $S_i$ and the previous summary vector $\sigma_{i-1}$ as the input. The language model is then applied over this input to obtain the updated summary vector and the token representation (right).
• Figure 3: Self-supervised Learning.
• Figure 4: Visualization of summary vectors.
• Figure 5: Visualization of summary vectors on OKTA.