Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Jacob Ziv's Individual-Sequence Approach to Information Theory

Neri Merhav

TL;DR

The paper surveys Jacob Ziv's individual-sequence approach and the LZ family of algorithms, arguing that this deterministic view enables universal data processing without assuming probabilistic source models. It develops the finite-state compressibility framework and the LZ complexity as individual-sequence analogues of entropy, and shows that LZ78 achieves a coding theorem and universality for stationary ergodic sources. Beyond compression, the article demonstrates how LZ-based methods—underpinned by Ziv's inequality and related constructs—yield asymptotically optimal solutions across diverse tasks, including hypothesis testing, model-order estimation, sequence divergence, universal decoding, encryption, gambling, prediction, filtering, guessing, and universal rate-distortion ensembles. The work emphasizes the broad theoretical and practical impact of incremental parsing as a unifying principle for statistics gathering and decision rules in information processing, with implications for future networked and multiuser settings.

Abstract

This article stands as a tribute to the enduring legacy of Jacob Ziv and his landmark contributions to information theory. Specifically, it delves into the groundbreaking individual-sequence approach -- a cornerstone of Ziv's academic pursuits. Together with Abraham Lempel, Ziv pioneered the renowned Lempel-Ziv (LZ) algorithm, a beacon of innovation in various versions. Beyond its original domain of universal data compression, this article underscores the broad utility of the individual-sequence approach and the LZ algorithm across a wide spectrum of problem areas. As we traverse through the forthcoming pages, it will also become evident how Ziv's visionary approach has left an indelible mark on my own research journey, as well as on those of numerous colleagues and former students. We shall explore, not only the technical power of the LZ algorithm, but also its profound impact on shaping the landscape of information theory and its applications.

On Jacob Ziv's Individual-Sequence Approach to Information Theory

TL;DR

The paper surveys Jacob Ziv's individual-sequence approach and the LZ family of algorithms, arguing that this deterministic view enables universal data processing without assuming probabilistic source models. It develops the finite-state compressibility framework and the LZ complexity as individual-sequence analogues of entropy, and shows that LZ78 achieves a coding theorem and universality for stationary ergodic sources. Beyond compression, the article demonstrates how LZ-based methods—underpinned by Ziv's inequality and related constructs—yield asymptotically optimal solutions across diverse tasks, including hypothesis testing, model-order estimation, sequence divergence, universal decoding, encryption, gambling, prediction, filtering, guessing, and universal rate-distortion ensembles. The work emphasizes the broad theoretical and practical impact of incremental parsing as a unifying principle for statistics gathering and decision rules in information processing, with implications for future networked and multiuser settings.

Abstract

This article stands as a tribute to the enduring legacy of Jacob Ziv and his landmark contributions to information theory. Specifically, it delves into the groundbreaking individual-sequence approach -- a cornerstone of Ziv's academic pursuits. Together with Abraham Lempel, Ziv pioneered the renowned Lempel-Ziv (LZ) algorithm, a beacon of innovation in various versions. Beyond its original domain of universal data compression, this article underscores the broad utility of the individual-sequence approach and the LZ algorithm across a wide spectrum of problem areas. As we traverse through the forthcoming pages, it will also become evident how Ziv's visionary approach has left an indelible mark on my own research journey, as well as on those of numerous colleagues and former students. We shall explore, not only the technical power of the LZ algorithm, but also its profound impact on shaping the landscape of information theory and its applications.
Paper Structure (14 sections, 28 equations, 3 figures)

This paper contains 14 sections, 28 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Modeling Approaches and Sequence Complexity
  3. Compression of Individual Sequences by FSM's - the LZ Algorithm
  4. The LZ Algorithm at the Service of Tasks Beyond Compression
  5. Hypothesis Testing and Model Order Estimation
  6. A Measure of Divergence between Sequences
  7. Universal Channel Decoding
  8. Encryption
  9. Gambling
  10. Prediction
  11. Filtering
  12. Guessing
  13. Universal Code Ensembles
  14. Summary and Outlook

Figures (3)

  • Figure 1: The hierarchy of classes of sources with various degrees of generality.
  • Figure 2: Finite-state encoder.
  • Figure 3: Physical system with an information reservoir