Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A faster algorithm for efficient longest common substring calculation for non-parametric entropy estimation in sequential data

Bridget Smart, Max Ward, Matthew Roughan

TL;DR

The paper addresses the computational bottleneck in non-parametric entropy estimation for sequential data, where the longest common substring (LCS) length is required for pattern-based entropy estimates. It introduces LCSFinder, an algorithm that achieves sub-quadratic performance by leveraging a sorted suffix array combined with a persistent binary search tree to answer relaxed and strict LCS queries on dynamically growing sequences, yielding a worst-case complexity of $O(N \log^2 N)$ for equal-length inputs. The authors provide theoretical analysis and extensive empirical validation on synthetic and real-world datasets, including a large social media corpus, demonstrating dramatic speedups over brute-force approaches while maintaining accuracy. They also release a standalone package and integrate the method into ProcessEntropy, broadening the practical applicability of non-parametric entropy estimation for large-scale, real-time signal processing and information-flow studies.

Abstract

Non-parametric entropy estimation on sequential data is a fundamental tool in signal processing, capturing information flow within or between processes to measure predictability, redundancy, or similarity. Methods based on longest common substrings (LCS) provide a non-parametric estimate of typical set size but are often inefficient, limiting use on real-world data. We introduce LCSFinder, a new algorithm that improves the worst-case performance of LCS calculations from cubic to log-linear time. Although built on standard algorithmic constructs - including sorted suffix arrays and persistent binary search trees - the details require care to provide the matches required for entropy estimation on dynamically growing sequences. We demonstrate that LCSFinder achieves dramatic speedups over existing implementations on real and simulated data, enabling entropy estimation at scales previously infeasible in practical signal processing.

A faster algorithm for efficient longest common substring calculation for non-parametric entropy estimation in sequential data

TL;DR

The paper addresses the computational bottleneck in non-parametric entropy estimation for sequential data, where the longest common substring (LCS) length is required for pattern-based entropy estimates. It introduces LCSFinder, an algorithm that achieves sub-quadratic performance by leveraging a sorted suffix array combined with a persistent binary search tree to answer relaxed and strict LCS queries on dynamically growing sequences, yielding a worst-case complexity of for equal-length inputs. The authors provide theoretical analysis and extensive empirical validation on synthetic and real-world datasets, including a large social media corpus, demonstrating dramatic speedups over brute-force approaches while maintaining accuracy. They also release a standalone package and integrate the method into ProcessEntropy, broadening the practical applicability of non-parametric entropy estimation for large-scale, real-time signal processing and information-flow studies.

Abstract

Non-parametric entropy estimation on sequential data is a fundamental tool in signal processing, capturing information flow within or between processes to measure predictability, redundancy, or similarity. Methods based on longest common substrings (LCS) provide a non-parametric estimate of typical set size but are often inefficient, limiting use on real-world data. We introduce LCSFinder, a new algorithm that improves the worst-case performance of LCS calculations from cubic to log-linear time. Although built on standard algorithmic constructs - including sorted suffix arrays and persistent binary search trees - the details require care to provide the matches required for entropy estimation on dynamically growing sequences. We demonstrate that LCSFinder achieves dramatic speedups over existing implementations on real and simulated data, enabling entropy estimation at scales previously infeasible in practical signal processing.
Paper Structure (9 sections, 4 equations, 2 figures)

This paper contains 9 sections, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Notation
  4. Entropy Calculation
  5. Existing work
  6. Calculating the longest common substring
  7. Experimental Results
  8. Validation on real and simulated datasets
  9. Conclusion

Figures (2)

  • Figure 1: Indices on the strings $S$ and $T$.
  • Figure 2: Empirically estimated complexity of the previously existing algorithm and that of LCSFinder as compared to the total length of both sequences ($N_T = N_S = N$). Note the log-log axes. The existing approach scales poorly as the sequence length increases due to its polynomial complexity. We can see that the LCSFinder algorithm achieves a performance bounded by the expected theoretical complexity. Similar performance is seen on the simulated and Twitter dataset, but LCSFinder's performance is closer for both.