Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Generalized Information Bottleneck Method: A Decision-Theoretic Perspective

Akira Kamatsuka, Takahiro Yoshida

TL;DR

This study derives an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem and derives a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information.

Abstract

The information bottleneck (IB) method seeks a compressed representation of data that preserves information relevant to a target variable for prediction while discarding irrelevant information from the original data. In its classical formulation, the IB method employs mutual information to evaluate the compression between the original and compressed data and the utility of the representation for the target variable. In this study, we investigate a generalized IB problem, where the evaluation of utility is based on the $\mathcal{H}$-mutual information that satisfies the concave (\texttt{CV}) and averaging (\texttt{AVG}) conditions. This class of information measures admits a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information. Based on this interpretation, we derive an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem.

A Generalized Information Bottleneck Method: A Decision-Theoretic Perspective

TL;DR

This study derives an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem and derives a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information.

Abstract

The information bottleneck (IB) method seeks a compressed representation of data that preserves information relevant to a target variable for prediction while discarding irrelevant information from the original data. In its classical formulation, the IB method employs mutual information to evaluate the compression between the original and compressed data and the utility of the representation for the target variable. In this study, we investigate a generalized IB problem, where the evaluation of utility is based on the -mutual information that satisfies the concave (\texttt{CV}) and averaging (\texttt{AVG}) conditions. This class of information measures admits a statistical decision-theoretic interpretation via its equivalence to the expected value of sample information. Based on this interpretation, we derive an alternating optimization algorithm to assess the tradeoff between compression and utility in the generalized IB problem.
Paper Structure (11 sections, 8 theorems, 38 equations, 2 figures, 1 algorithm)

This paper contains 11 sections, 8 theorems, 38 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. A Property of Shannon MI and EVSI
  4. Definition and Properties of $\mathcal{H}$-MI
  5. A Generalized Information Bottleneck Problem
  6. Formulation of the Generalized IB problem
  7. Derivation of an Alternating Optimization Algorithm
  8. Discussion on the Convergence of the Alternating Optimization Algorithm
  9. Numerical Examples
  10. Conclusion
  11. Proof of Proposition \ref{['prop:update_formulae_SDT_IBM']}

Key Result

Proposition 1

Let $(X,T)\sim p_{X,T}$. Then, where the minimum is achieved when $q_T=p_T$.

Figures (2)

  • Figure 1: A system model for the information bottleneck problem incorporating a decision-maker.
  • Figure 2: Tradeoff curves of the generalized IB problem when adopting the Shannon MI $I(Y;T)$ (blue dots) and $\mathrm{EVSI}^{\ell_{\mathrm{sq}}}(Y;T)$ (red dots) as utility measures.

Theorems & Definitions (34)

  • Proposition 1: Cover:2006:EIT:1146355
  • Proposition 2: ghosh2007introduction
  • Example 1: Point estimation of $Y$
  • Example 2: Decision on the probability distribution of $Y$
  • Definition 1: Proper scoring rule
  • Example 3
  • Remark 1
  • Definition 2: EVSI raiffa1961applied
  • Example 4
  • Example 5
  • ...and 24 more