DeLog: An Efficient Log Compression Framework with Pattern Signature Synthesis

TL;DR

This work rethinks log compression by showing that parsing accuracy does not reliably predict compression gains. It introduces DeLog, a pattern-based grouper that synthesizes token-level signatures from intrinsic structure and external context to create homogeneous data streams amenable to specialized encoding. Across 16 public benchmarks and 10 production datasets, DeLog achieves state-of-the-art compression ratios and significantly faster compression/decompression speeds, with a lightweight DeLog-L variant offering exceptional decompression throughput. The approach offers robust performance on real-world logs and opens avenues for semantic-enhanced pattern grouping in future work.

Abstract

Parser-based log compression, which separates static templates from dynamic variables, is a promising approach to exploit the unique structure of log data. However, its performance on complex production logs is often unsatisfactory. This performance gap coincides with a known degradation in the accuracy of its core log parsing component on such data, motivating our investigation into a foundational yet unverified question: does higher parsing accuracy necessarily lead to better compression ratio? To answer this, we conduct the first empirical study quantifying this relationship and find that a higher parsing accuracy does not guarantee a better compression ratio. Instead, our findings reveal that compression ratio is dictated by achieving effective pattern-based grouping and encoding, i.e., the partitioning of tokens into low entropy, highly compressible groups. Guided by this insight, we design DeLog, a novel log compressor that implements a Pattern Signature Synthesis mechanism to achieve efficient pattern-based grouping. On 16 public and 10 production datasets, DeLog achieves state-of-the-art compression ratio and speed.

Figures (11)

• Figure 1: Log Structure: High redundancy stems from the repetition of static templates, while dynamic variables introduce variability.
• Figure 2: The General Steps of Parser-based Log Compressor.
• Figure 3: Performance of Existing Log Compressors on Production Logs: the four log datasets (A-D) are from ByteDance's Volcano Engine, each containing over 1m lines. ERROR indicates a catastrophic failure (e.g., a segmentation fault) that prevented lossless compression. All compressors run with 4 threads and the highest compression level on 100k-line blocks.
• Figure 4: Spearman Correlation Between Log Parsing Accuracy Metrics and CR (Aggregated Across 12 Datasets from Loghub 2.0).
• Figure 5: The Overview of DeLog.