Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BAT-LZ Out of Hell

Zsuzsanna Lipták, Francesco Masillo, Gonzalo Navarro

TL;DR

This work introduces BAT-LZ, a bounded-access variant of Lempel-Ziv that guarantees $O(c)$ access time per symbol by bounding reference chains with a parameter $c$, making it attractive for compressed indices. It develops a Greedy BAT-LZ parser with linear space and $O(n\log^3 n)$ time via a geometric data structure supporting 5-sided queries, plus a space-time trade-off to $O(n\log n)$ space with $O(n\log^2 n)$ time; it also presents a Minmax parser using an enhanced suffix tree to optimize source choices, and a Greedier parser that combines both to achieve near-longest-phrase parsing with superior compression. Experiments show that on repetitive texts the parsers attain access costs close to $\log_2 n$ with compression near classical LZ, often with only small overhead in phrase count. The study highlights practical pathways and open problems for bounded-access parsing, including potential integrations with LZ-based indexes and extensions to other LZ variants.

Abstract

Despite consistently yielding the best compression on repetitive text collections, the Lempel-Ziv parsing has resisted all attempts at offering relevant guarantees on the cost to access an arbitrary symbol. This makes it less attractive for use on compressed self-indexes and other compressed data structures. In this paper we introduce a variant we call BAT-LZ (for Bounded Access Time Lempel-Ziv) where the access cost is bounded by a parameter given at compression time. We design and implement a linear-space algorithm that, in time $O(n\log^3 n)$, obtains a BAT-LZ parse of a text of length $n$ by greedily maximizing each next phrase length. The algorithm builds on a new linear-space data structure that solves 5-sided orthogonal range queries in rank space, allowing updates to the coordinate where the one-sided queries are supported, in $O(\log^3 n)$ time for both queries and updates. This time can be reduced to $O(\log^2 n)$ if $O(n\log n)$ space is used. We design a second algorithm that chooses the sources for the phrases in a clever way, using an enhanced suffix tree, albeit no longer guaranteeing longest possible phrases. This algorithm is much slower in theory, but in practice it is comparable to the greedy parser, while achieving significantly superior compression. We then combine the two algorithms, resulting in a parser that always chooses the longest possible phrases, and the best sources for those. Our experimentation shows that, on most repetitive texts, our algorithms reach an access cost close to $\log_2 n$ on texts of length $n$, while incurring almost no loss in the compression ratio when compared with classical LZ-compression. Several open challenges are discussed at the end of the paper.

BAT-LZ Out of Hell

TL;DR

This work introduces BAT-LZ, a bounded-access variant of Lempel-Ziv that guarantees access time per symbol by bounding reference chains with a parameter , making it attractive for compressed indices. It develops a Greedy BAT-LZ parser with linear space and time via a geometric data structure supporting 5-sided queries, plus a space-time trade-off to space with time; it also presents a Minmax parser using an enhanced suffix tree to optimize source choices, and a Greedier parser that combines both to achieve near-longest-phrase parsing with superior compression. Experiments show that on repetitive texts the parsers attain access costs close to with compression near classical LZ, often with only small overhead in phrase count. The study highlights practical pathways and open problems for bounded-access parsing, including potential integrations with LZ-based indexes and extensions to other LZ variants.

Abstract

Despite consistently yielding the best compression on repetitive text collections, the Lempel-Ziv parsing has resisted all attempts at offering relevant guarantees on the cost to access an arbitrary symbol. This makes it less attractive for use on compressed self-indexes and other compressed data structures. In this paper we introduce a variant we call BAT-LZ (for Bounded Access Time Lempel-Ziv) where the access cost is bounded by a parameter given at compression time. We design and implement a linear-space algorithm that, in time , obtains a BAT-LZ parse of a text of length by greedily maximizing each next phrase length. The algorithm builds on a new linear-space data structure that solves 5-sided orthogonal range queries in rank space, allowing updates to the coordinate where the one-sided queries are supported, in time for both queries and updates. This time can be reduced to if space is used. We design a second algorithm that chooses the sources for the phrases in a clever way, using an enhanced suffix tree, albeit no longer guaranteeing longest possible phrases. This algorithm is much slower in theory, but in practice it is comparable to the greedy parser, while achieving significantly superior compression. We then combine the two algorithms, resulting in a parser that always chooses the longest possible phrases, and the best sources for those. Our experimentation shows that, on most repetitive texts, our algorithms reach an access cost close to on texts of length , while incurring almost no loss in the compression ratio when compared with classical LZ-compression. Several open challenges are discussed at the end of the paper.
Paper Structure (9 sections, 1 theorem, 5 figures, 1 table)

This paper contains 9 sections, 1 theorem, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Basic Data Structures
  3. The Lempel-Ziv (LZ) Parsing and its Bounded Version (BAT-LZ)
  4. A Greedy Parsing Algorithm for BAT-LZ
  5. A Geometric Data Structure
  6. The Minmax Parsing Algorithm
  7. The Greedier Parser: Combining Greedy with Minmax
  8. Experiments
  9. Discussion and Future Work

Key Result

Theorem 5

A Greedy BAT-LZ parse of a text $T[1\mathinner{.\,.} n]$ can be computed using $O(n)$ space and $O(n\log^3 n)$ time, or $O(n\log n)$ space and $O(n\log^2 n)$ time.

Figures (5)

  • Figure 1: General scheme of our translation of queries onto a 3-dimensional data structure.
  • Figure 2: On the left, we reach a candidate area $[sp_3,ep_3]$ of the wavelet matrix and must obtain its maximum $D$ value using the (dynamic) RMQ data structure for $D_3$. The tree $H_3$ for this RMQ structure is shown on the right. Arrows point to the child holding the maximum value in $D_3$. Blue diamonds are the roots $v_3^1,\ldots,v_3^4$ of the subtrees that cover the query area $[sp_3,ep_3]$ and red circles are the candidates in the range. The left plot shows how we find the actual value of one of those circles by tracking it down in the wavelet matrix.
  • Figure 3: Example of the Minmax algorithm using the suffix tree of $T={\tt alabaralalabarda\$}$. The vertical bars are for delimiting already parsed phrases. See Example \ref{['ex:greedier']} for more details.
  • Figure 4: Overhead factor of number of BAT-LZ versus LZ phrases as a function of the maximum length $c$ of a chain, for our different BAT-LZ parsers and a balanced grammar.
  • Figure 5: Left: detail of Fig. \ref{['fig:plot']}, for the Greedier BAT-LZ parser, focusing on the overheads below 10% over the LZ parse. Right: a comparison of histograms with shared $x$ and $y$ axis representing the chain length values on leaders; LZ on top and Greedier BAT-LZ with $c = 20$ on the bottom.

Theorems & Definitions (5)

  • Definition 1
  • Definition 2: Chain length
  • Definition 3
  • Theorem 5
  • Example 6