LogPrism: Unifying Structure and Variable Encoding for Effective Log Compression

TL;DR

LogPrism addresses the inefficiency of parse-then-compress pipelines by unifying structure and variable encoding through a Unified Redundancy Tree. Its three-stage hierarchical approach first builds a structural skeleton, then mines structure-variable co-occurrence patterns, and finally processes residual data, enabling single identifiers to represent highly correlated token sequences. Empirical evaluation over 16 LogHub datasets shows LogPrism achieves state-of-the-art compression ratios on most datasets and attains high throughput, with a parallel variant delivering even greater speeds. The key contribution is the co-design of parsing and compression, preserving deep contextual redundancies across templates and parameters, enabling robust, scalable log compression for large-scale systems.

Abstract

The prevailing "parse-then-compress" paradigm in log compression fundamentally limits effectiveness by treating log parsing and compression as isolated objectives. While parsers prioritize semantic accuracy (i.e., event identification), they often obscure deep correlations between static templates and dynamic variables that are critical for storage efficiency. In this paper, we investigate this misalignment through a comprehensive empirical study and propose LogPrism, a framework that bridges the gap via unified redundancy encoding. Rather than relying on a rigid pre-parsing step, LogPrism dynamically integrates structural extraction with variable encoding by constructing a Unified Redundancy Tree (URT). This hierarchical approach effectively mines "structure+variable" co-occurrence patterns, capturing deep contextual redundancies while accelerating processing through pre-emptive pattern encoding. Extensive experiments on 16 benchmark datasets confirm that LogPrism establishes a new state-of-the-art. It achieves the highest compression ratio on 13 datasets, surpassing leading baselines by margins of 4.7% to 80.9%, while delivering superior throughput at 29.87 MB/s (1.68$\times$~43.04$\times$ faster than competitors). Moreover, when configured in single-archive mode to maximize global pattern discovery, LogPrism outperforms the best baseline by 19.39% in compression ratio while maintaining a 2.62$\times$ speed advantage.

Figures (11)

• Figure 1: The General Workflow of Parser-based Log Compression
• Figure 2: Impact of Log Parser Selection on Downstream Compression Ratio across Different Datasets
• Figure 3: A 12-line Log Snippet as a Running Example
• Figure 4: An Overview of the LogPrism Compression Framework
• Figure 5: The Pre-processing Pipeline for Log Entry L5