Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Engineering Fast and Space-Efficient Recompression from SLP-Compressed Text

Ankith Reddy Adudodla, Dominik Kempa

TL;DR

This work tackles the practical bottleneck of constructing recompression RLSLPs for highly repetitive text by presenting the first end-to-end recompression pipeline that operates in compressed time. The approach starts from an approximate LZ77-like parsing and alternates Block Compression and Pair Compression on SLP/SLG representations, achieving a compact, local-consistent RLSLP with modest peak RAM. Empirically, the pipeline delivers up to $46×$ speedups and $17×$ RAM reductions on 4 GiB inputs, with strong scalability to 8 GiB datasets and better memory behavior than uncompressed baselines. The results demonstrate that compressed-time computation is a viable and scalable path for fast index construction on large, repetitive corpora, enabling practical deployment of advanced text indexes.

Abstract

Compressed indexing enables powerful queries over massive and repetitive textual datasets using space proportional to the compressed input. While theoretical advances have led to highly efficient index structures, their practical construction remains a bottleneck, especially for complex components like recompression RLSLP, a grammar-based representation crucial for building powerful text indexes that support widely used suffix and LCP array queries. In this work, we present the first implementation of recompression RLSLP construction that runs in compressed time, operating on an LZ77-like approximation of the input. Compared to state-of-the-art uncompressed-time methods, our approach achieves up to $46\times$ speedup and $17\times$ lower RAM usage on large, repetitive inputs. These gains unlock scalability to larger datasets and affirm compressed computation as a practical path forward for fast index construction.

Engineering Fast and Space-Efficient Recompression from SLP-Compressed Text

TL;DR

This work tackles the practical bottleneck of constructing recompression RLSLPs for highly repetitive text by presenting the first end-to-end recompression pipeline that operates in compressed time. The approach starts from an approximate LZ77-like parsing and alternates Block Compression and Pair Compression on SLP/SLG representations, achieving a compact, local-consistent RLSLP with modest peak RAM. Empirically, the pipeline delivers up to speedups and RAM reductions on 4 GiB inputs, with strong scalability to 8 GiB datasets and better memory behavior than uncompressed baselines. The results demonstrate that compressed-time computation is a viable and scalable path for fast index construction on large, repetitive corpora, enabling practical deployment of advanced text indexes.

Abstract

Compressed indexing enables powerful queries over massive and repetitive textual datasets using space proportional to the compressed input. While theoretical advances have led to highly efficient index structures, their practical construction remains a bottleneck, especially for complex components like recompression RLSLP, a grammar-based representation crucial for building powerful text indexes that support widely used suffix and LCP array queries. In this work, we present the first implementation of recompression RLSLP construction that runs in compressed time, operating on an LZ77-like approximation of the input. Compared to state-of-the-art uncompressed-time methods, our approach achieves up to speedup and lower RAM usage on large, repetitive inputs. These gains unlock scalability to larger datasets and affirm compressed computation as a practical path forward for fast index construction.

Paper Structure

This paper contains 28 sections, 3 theorems, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our Results
  3. Preliminaries
  4. Strings
  5. LZ77 Compression
  6. Grammar Compression
  7. Overview of Recompression
  8. Recompression in Compressed Time
  9. Implementation Details
  10. Representing Intermediate Strings
  11. Block Compression (BComp)
  12. Computing Left and Right Runs
  13. Supporting Grammar Updates
  14. Pair Compression (PComp)
  15. Experiments
  16. ...and 13 more sections

Key Result

Theorem 2.1

Given the non-overlapping LZ77 factorization of a string $T$ of length $n$, we can in $\mathcal{O}(z_{\rm no}(T) \log n)$ time construct an SLP $G$ such that $L(G) = \{T\}$.

Figures (3)

  • Figure 5.1: Impact of block size on compression pipeline components (dataset: Y4M5).
  • Figure 5.2: Runtime breakdown of major pipeline components across block sizes. Only stages with non-negligible runtime are shown.
  • Figure 5.3: Peak RAM usage of each compressed pipeline component across block sizes.

Theorems & Definitions (3)

  • Theorem 2.1: Rytter03charikar
  • Theorem 2.2: Gawrychowski11, resolution
  • Theorem 2.3: collapsing