Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting the Folklore Algorithm for Random Access to Grammar-Compressed Strings

Alan M. Cleary, Joseph Winjum, Jordan Dood, Shunsuke Inenaga

TL;DR

The paper addresses practical random access on grammar-compressed strings and notes that prior results either require CNF conversion or balancing. It introduces FRAS, a modified folklore algorithm that operates directly on any grammar by sorting non-terminals by expansion length and using sparse rank-select bitvectors to track rule lengths and start positions, achieving strong space efficiency and near-\(O(\log n)\) time on balanced grammars. The authors demonstrate substantial space savings and competitive runtimes across Pizza&Chili and pangenome datasets, including MR-RePair grammars, underscoring the method's practicality. Overall, the work provides a grammar-agnostic random-access approach with significant real-world impact for large compressed texts like pangenomes, and outlines future directions to further reduce space without sacrificing speed.

Abstract

Grammar-based compression is a widely-accepted model of string compression that allows for efficient and direct manipulations on the compressed data. Most, if not all, such manipulations rely on the primitive \emph{random access} queries, a task of quickly returning the character at a specified position of the original uncompressed string without explicit decompression. While there are advanced data structures for random access to grammar-compressed strings that guarantee theoretical query time and space bounds, little has been done for the \emph{practical} perspective of this important problem. In this paper, we revisit a well-known folklore random access algorithm for grammars in the Chomsky normal form, modify it to work directly on general grammars, and show that this modified version is fast and memory efficient in practice.

Revisiting the Folklore Algorithm for Random Access to Grammar-Compressed Strings

TL;DR

The paper addresses practical random access on grammar-compressed strings and notes that prior results either require CNF conversion or balancing. It introduces FRAS, a modified folklore algorithm that operates directly on any grammar by sorting non-terminals by expansion length and using sparse rank-select bitvectors to track rule lengths and start positions, achieving strong space efficiency and near-\(O(\log n)\) time on balanced grammars. The authors demonstrate substantial space savings and competitive runtimes across Pizza&Chili and pangenome datasets, including MR-RePair grammars, underscoring the method's practicality. Overall, the work provides a grammar-agnostic random-access approach with significant real-world impact for large compressed texts like pangenomes, and outlines future directions to further reduce space without sacrificing speed.

Abstract

Grammar-based compression is a widely-accepted model of string compression that allows for efficient and direct manipulations on the compressed data. Most, if not all, such manipulations rely on the primitive \emph{random access} queries, a task of quickly returning the character at a specified position of the original uncompressed string without explicit decompression. While there are advanced data structures for random access to grammar-compressed strings that guarantee theoretical query time and space bounds, little has been done for the \emph{practical} perspective of this important problem. In this paper, we revisit a well-known folklore random access algorithm for grammars in the Chomsky normal form, modify it to work directly on general grammars, and show that this modified version is fast and memory efficient in practice.
Paper Structure (15 sections, 1 figure, 6 tables, 2 algorithms)

This paper contains 15 sections, 1 figure, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Strings
  4. Grammar-based compression
  5. Algorithms
  6. The Folklore Algorithm
  7. Modified Folklore Algorithm
  8. Results
  9. Implementation
  10. Experiments
  11. Pizza&Chili
  12. Pangenomes
  13. Conclusion
  14. Acknowledgments.
  15. Appendix

Figures (1)

  • Figure 1: Example grammars and data structures for the modified folklore algorithm on string $T = \text{agagcgagagcgcgc}$. (a) and (b) depict the derivation trees for a RePair grammar $\mathcal{G}_\mathrm{RePair}$ and an SLP grammar $\mathcal{G}_\mathrm{SLP}$, respectively. (c) depicts the data structures used by our modified version of the folklore random access algorithm. Note that the subscript $i$ for each non-terminal $X_i$ conveys the array index the non-terminal in the array of arrays representation of the grammars. For simplicity, terminal characters are used directly without a proxy non-terminal character.