Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Toolbox for Refined Information-Theoretic Analyses with Applications

Neri Merhav, Nir Weinberger

TL;DR

This monograph offers a toolbox of mathematical techniques, which have been effective and widely applicable in information-theoretic analysis, which allow to refine the results of the method of types, and are capable of obtaining more precise results.

Abstract

This monograph offers a toolbox of mathematical techniques, which have been effective and widely applicable in information-theoretic analysis. The first tool is a generalization of the method of types to Gaussian settings, and then to general exponential families. The second tool is Laplace and saddle-point integration, which allow to refine the results of the method of types, and are capable of obtaining more precise results. The third is the type class enumeration method, a principled method to evaluate the exact random-coding exponent of coded systems, which results in the best known exponent in various problem settings. The fourth subset of tools aimed at evaluating the expectation of non-linear functions of random variables, either via integral representations, or by a refinement of Jensen's inequality via change-of-measure, by complementing Jensen's inequality with a reversed inequality, or by a class of generalized Jensen's inequalities that are applicable for functions beyond convex/concave. Various application examples of all these tools are provided along this monograph.

A Toolbox for Refined Information-Theoretic Analyses with Applications

TL;DR

This monograph offers a toolbox of mathematical techniques, which have been effective and widely applicable in information-theoretic analysis, which allow to refine the results of the method of types, and are capable of obtaining more precise results.

Abstract

This monograph offers a toolbox of mathematical techniques, which have been effective and widely applicable in information-theoretic analysis. The first tool is a generalization of the method of types to Gaussian settings, and then to general exponential families. The second tool is Laplace and saddle-point integration, which allow to refine the results of the method of types, and are capable of obtaining more precise results. The third is the type class enumeration method, a principled method to evaluate the exact random-coding exponent of coded systems, which results in the best known exponent in various problem settings. The fourth subset of tools aimed at evaluating the expectation of non-linear functions of random variables, either via integral representations, or by a refinement of Jensen's inequality via change-of-measure, by complementing Jensen's inequality with a reversed inequality, or by a class of generalized Jensen's inequalities that are applicable for functions beyond convex/concave. Various application examples of all these tools are provided along this monograph.
Paper Structure (47 sections, 4 theorems, 348 equations, 1 figure)

This paper contains 47 sections, 4 theorems, 348 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Extension of the Method of Types to Continuous Alphabets
  3. Introduction
  4. Various Definitions of Type Classes
  5. Simple Gaussian Types
  6. More Refined Gaussian Types
  7. Conditional Gaussian Types
  8. Gauss--Markov Types
  9. Types Classes Pertaining to Exponential Families
  10. Applications
  11. The Laplace Method of Integration and the Saddle-Point Method
  12. Introduction
  13. The Laplace Method of Integration
  14. The Saddle-Point Method
  15. Applications
  16. ...and 32 more sections

Key Result

Theorem 1

Assume that $N\sim\text{Binomial}(e^{nA},e^{-nB})$ and $\lambda\in\mathbbm{R}$. Then, the upper tail is and the lower tail is

Figures (1)

  • Figure 1: A path ${\cal P}$ from $A$ to $B$, passing via $z_{0}$ along the axis.

Theorems & Definitions (34)

  • Example 1
  • Example 2
  • Example 3
  • Example 4
  • Example 5
  • Example 6
  • Example 7: Universal coding
  • Example 8: Universal coding revisited
  • Example 9: Extreme Value Statistics
  • Example 10: The Stirling formula
  • ...and 24 more