Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Sample Complexity of Lossless Data Compression

Terence Viaud, Ioannis Kontoyiannis

TL;DR

A new framework is introduced for examining and evaluating the fundamental limits of lossless data compression, that emphasizes genuinely non-asymptotic results and bounds on the sample complexity of universal data compression are developed for arbitrary families of memoryless sources.

Abstract

A new framework is introduced for examining and evaluating the fundamental limits of lossless data compression, that emphasizes genuinely non-asymptotic results. The {\em sample complexity} of compressing a given source is defined as the smallest blocklength at which it is possible to compress that source at a specified rate and to within a specified excess-rate probability. This formulation parallels corresponding developments in statistics and computer science, and it facilitates the use of existing results on the sample complexity of various hypothesis testing problems. For arbitrary sources, the sample complexity of general variable-length compressors is shown to be tightly coupled with the sample complexity of prefix-free codes and fixed-length codes. For memoryless sources, it is shown that the sample complexity is characterized not by the source entropy, but by its Rényi entropy of order~$1/2$. Nonasymptotic bounds on the sample complexity are obtained, with explicit constants. Generalizations to Markov sources are established, showing that the sample complexity is determined by the source's Rényi entropy rate of order~$1/2$. Finally, bounds on the sample complexity of universal data compression are developed for arbitrary families of memoryless sources. There, the sample complexity is characterized by the minimum Rényi divergence of order~$1/2$ between elements of the family and the uniform distribution. The connection of this problem with identity testing and with the associated separation rates is explored and discussed.

The Sample Complexity of Lossless Data Compression

TL;DR

A new framework is introduced for examining and evaluating the fundamental limits of lossless data compression, that emphasizes genuinely non-asymptotic results and bounds on the sample complexity of universal data compression are developed for arbitrary families of memoryless sources.

Abstract

A new framework is introduced for examining and evaluating the fundamental limits of lossless data compression, that emphasizes genuinely non-asymptotic results. The {\em sample complexity} of compressing a given source is defined as the smallest blocklength at which it is possible to compress that source at a specified rate and to within a specified excess-rate probability. This formulation parallels corresponding developments in statistics and computer science, and it facilitates the use of existing results on the sample complexity of various hypothesis testing problems. For arbitrary sources, the sample complexity of general variable-length compressors is shown to be tightly coupled with the sample complexity of prefix-free codes and fixed-length codes. For memoryless sources, it is shown that the sample complexity is characterized not by the source entropy, but by its Rényi entropy of order~. Nonasymptotic bounds on the sample complexity are obtained, with explicit constants. Generalizations to Markov sources are established, showing that the sample complexity is determined by the source's Rényi entropy rate of order~. Finally, bounds on the sample complexity of universal data compression are developed for arbitrary families of memoryless sources. There, the sample complexity is characterized by the minimum Rényi divergence of order~ between elements of the family and the uniform distribution. The connection of this problem with identity testing and with the associated separation rates is explored and discussed.
Paper Structure (21 sections, 19 theorems, 124 equations)

This paper contains 21 sections, 19 theorems, 124 equations.

Table of Contents

  1. Introduction
  2. Lossless compressors
  3. Asymptotic approximations
  4. Sample complexity
  5. Outline of main results
  6. History and general remarks on sample complexity
  7. Preliminaries
  8. Sample complexity of fixed-length compression
  9. Fixed-length codes and sample complexity
  10. Memoryless sources
  11. Sample complexity of variable-length compression
  12. Variable-length codes and sample complexity
  13. Memoryless sources
  14. Markov sources
  15. Rényi divergence rate and irreducible Markov sources
  16. ...and 6 more sections

Key Result

Proposition 2.1

Suppose $P,Q$ are arbitrary p.m.f.s on a finite alphabet $A$. Then, for any $n\geq 1$, where $P^n,Q^n$ denote the corresponding product p.m.f.s on $A^n$.

Theorems & Definitions (19)

  • Proposition 2.1: Tensorization of $D_{1/2}(P\|Q)$
  • Proposition 2.2: Rényi divergence and total variation
  • Proposition 3.1: Le Cam's lemma
  • Theorem 3.2: Fixed-length sample complexity of memoryless sources
  • Theorem 4.1: Fixed- vs. variable-length sample complexity
  • Theorem 4.2: Variable-length vs. prefix-free sample complexity
  • Theorem 4.3: Variable-length sample complexity of memoryless sources
  • Proposition 5.1: Rényi divergence of Markov chains
  • Theorem 5.2: Sample complexity of irreducible Markov sources
  • Theorem 5.3: Sample complexity of symmetric Markov sources
  • ...and 9 more