LZD-style Compression Scheme with Truncation and Repetitions
Linus Götz, Dominik Köppl
TL;DR
The authors address the inefficiencies and limited compressibility of LZD by introducing LZD+ for online linear-time compression using truncated references, and LZDR, which adds a repetition-based factorization rule to further improve compressibility. LZD+ employs an augmented radix trie to achieve $O(n)$ expected time (or $O(n \log \sigma)$ worst-case with deterministic lookups) and $O(z)$ space, while LZDR extends this with a repetition mechanism and an offline $O(n)$ algorithm aided by an $LCE$ data structure. Three flexible parsing variants for LZDR further reduce factor counts in practice, with benchmarks showing superior factor reduction compared with LZD and LZW-based methods on standard corpora, though the advantage may be small on very large datasets. The work demonstrates practical gains in factor efficiency and offers a foundation for online/offline, prefix-closed, repetition-enabled compression schemes with notable theoretical and empirical contributions to LZ-family methods.
Abstract
Lempel-Ziv-Double (LZD) is a variation of the LZ78 compression scheme that achieves better compression on repetitive datasets. Nevertheless, prior research has identified computational inefficiencies and a weakness in its compressibility for certain datasets. In this paper, we introduce LZD+, an enhancement of LZD, which enables expected linear-time online compression by allowing truncated references. To avoid the compressibility weakness exhibited by a lower bound example, we propose LZDR (LZD-runlength compressed), a further enhancement on top of LZD+, which introduces a repetition-based factorization rule while maintaining linear expected time complexity. The both time bounds can be de-randomized by a lookup data structure like a balanced search tree with a logarithmic dependency on the alphabet size. Additionally, we present three flexible parsing variants of LZDR that yield fewer factors in practice. Comprehensive benchmarking on standard corpora reveals that LZD+, LZDR, and its flexible variants outperform existing LZ-based methods in the number of factors while keeping competitive runtime efficiency. However, we note that the difference in the number of factors becomes marginal for large datasets like those of the Pizza&Chili corpus.