Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropies, cross-entropies and Rényi divergence: sharp three-term inequalities for probability density functions

Razvan Gabriel Iagar, David Puertas-Centeno

Abstract

A new sharp inequality featuring the differential Rényi entropy, the Rényi divergence and the Rényi cross-entropy of a pair of probability density functions is established. The equality is reached when one of the probability density function is an escort density of the other. This inequality is applied, together with a general framework of a pair of transformations reciprocal to each other, to derive a number of further inequalities involving both classical and new informational functionals. A remarkable fact is that, in all these inequalities, the Rényi divergence of two probability density functions is sharply bounded by quotients of informational functionals of cross-type and single type. More precisely, we derive sharp inequalities composed by relative and cross versions of the absolute moments, or of the Fisher information measures (among others), and involving two and three probability density functions.

Entropies, cross-entropies and Rényi divergence: sharp three-term inequalities for probability density functions

Abstract

A new sharp inequality featuring the differential Rényi entropy, the Rényi divergence and the Rényi cross-entropy of a pair of probability density functions is established. The equality is reached when one of the probability density function is an escort density of the other. This inequality is applied, together with a general framework of a pair of transformations reciprocal to each other, to derive a number of further inequalities involving both classical and new informational functionals. A remarkable fact is that, in all these inequalities, the Rényi divergence of two probability density functions is sharply bounded by quotients of informational functionals of cross-type and single type. More precisely, we derive sharp inequalities composed by relative and cross versions of the absolute moments, or of the Fisher information measures (among others), and involving two and three probability density functions.
Paper Structure (10 sections, 9 theorems, 104 equations)

This paper contains 10 sections, 9 theorems, 104 equations.

Table of Contents

  1. Introduction
  2. An inequality involving the Rényi entropy, the Rényi divergence and the Rényi cross-entropy
  3. A brief review on informational measures
  4. The main inequality
  5. Applications to further inequalities
  6. General framework
  7. Diferential-escort transformation
  8. Relative differential-escort transformation
  9. Biparametric down transformation
  10. Up transformation

Key Result

Theorem 2.1

Let $\alpha,\beta,\gamma$ be three real numbers satisfying Eq. eq:relation. If $\alpha>\beta$ then In the opposite case $\alpha<\beta$ the inequality is reversed. Moreover, the equality holds if and only if

Theorems & Definitions (30)

  • Theorem 2.1
  • proof
  • Proposition 3.1
  • proof
  • Definition 3.1
  • Lemma 3.1
  • proof
  • Theorem 3.1
  • proof
  • Definition 3.2
  • ...and 20 more