The information bottleneck method

Naftali Tishby; Fernando C. Pereira; William Bialek

Paper

The information bottleneck method

Abstract

We define the relevant information in a signal

as being the information that this signal provides about another signal

. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal

requires more than just predicting

, it also requires specifying which features of

play a role in the prediction. We formalize this problem as that of finding a short code for

that preserves the maximum information about

. That is, we squeeze the information that

provides about

through a `bottleneck' formed by a limited set of codewords

. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure

emerges from the joint statistics of

and

. This approach yields an exact set of self consistent equations for the coding rules

and

. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.