Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Similarity-Sensitive Entropy: Induced Kernels and Data-Processing Inequalities

Joseph Samuel Miller

TL;DR

This work works in the general measure-theoretic setting of kernelled probability spaces introduced by Leinster and Roff, and develops basic structural properties of an entropy functional that is sensitive to a prescribed similarity structure on a state space.

Abstract

We study an entropy functional $H_K$ that is sensitive to a prescribed similarity structure on a state space. For finite spaces, $H_K$ coincides with the order-1 similarity-sensitive entropy of Leinster and Cobbold. We work in the general measure-theoretic setting of kernelled probability spaces $(Ω,μ,K)$ introduced by Leinster and Roff, and develop basic structural properties of $H_K$. Our main results concern the behavior of $H_K$ under coarse-graining. For a measurable map $f:Ω\to Y$ and input law $μ$, we define a law-induced kernel on $Y$ whose pullback minimally dominates $K$, and show that it yields a coarse-graining inequality and a data-processing inequality for $H_K$, for both deterministic maps and general Markov kernels. We also introduce conditional similarity-sensitive entropy and an associated mutual information, and compare their behavior to the classical Shannon case.

Similarity-Sensitive Entropy: Induced Kernels and Data-Processing Inequalities

TL;DR

This work works in the general measure-theoretic setting of kernelled probability spaces introduced by Leinster and Roff, and develops basic structural properties of an entropy functional that is sensitive to a prescribed similarity structure on a state space.

Abstract

We study an entropy functional that is sensitive to a prescribed similarity structure on a state space. For finite spaces, coincides with the order-1 similarity-sensitive entropy of Leinster and Cobbold. We work in the general measure-theoretic setting of kernelled probability spaces introduced by Leinster and Roff, and develop basic structural properties of . Our main results concern the behavior of under coarse-graining. For a measurable map and input law , we define a law-induced kernel on whose pullback minimally dominates , and show that it yields a coarse-graining inequality and a data-processing inequality for , for both deterministic maps and general Markov kernels. We also introduce conditional similarity-sensitive entropy and an associated mutual information, and compare their behavior to the classical Shannon case.
Paper Structure (62 sections, 36 theorems, 206 equations)

This paper contains 62 sections, 36 theorems, 206 equations.

Table of Contents

  1. Introduction
  2. Notation and conventions.
  3. Similarity--Sensitive Entropy
  4. Discrete similarity--sensitive entropy
  5. Partition kernels and coarse variables (finite case)
  6. General kernelled probability spaces
  7. Isomorphisms and uniform representations
  8. Deterministic Coarse--Graining and Data--Processing
  9. Deterministic coarse--graining in the discrete case
  10. Induced coarse--graining kernels and back--composition in the discrete case
  11. Coarse--graining inequality
  12. Minimality/Uniqueness of the fiberwise max rule
  13. Deterministic coarse--graining on general probability spaces
  14. Setup and induced kernels
  15. Disintegration along $f$ (used only for a representation formula).
  16. ...and 47 more sections

Key Result

Proposition 2.6

Let $K$ be a partition kernel on $\mathcal{X}$ with classes $\{C_j\}$ and associated coarse variable $Z$. Then where $H(Z)$ is the Shannon entropy of $Z$.

Theorems & Definitions (111)

  • Definition 2.1: Similarity matrix on a finite set
  • Definition 2.2: Similarity--sensitive entropy in the discrete case
  • Remark 2.3
  • Definition 2.4: Partition kernel
  • Definition 2.5: Coarse variable associated to a partition kernel
  • Proposition 2.6
  • proof
  • Definition 2.7: Kernel on a probability space
  • Remark 2.8
  • Definition 2.9: Similarity--sensitive entropy on a probability space
  • ...and 101 more