Ambiguous signals and efficient codes

TL;DR

This work considers a scenario in which many sensors respond to a shared signal, each with limited information capacity, and asks that the outputs together convey as much information as possible about an underlying relevant variable.

Abstract

In many biological networks the responses of individual elements are ambiguous. We consider a scenario in which many sensors respond to a shared signal, each with limited information capacity, and ask that the outputs together convey as much information as possible about an underlying relevant variable. In a low noise limit where we can make analytic progress, we show that individually ambiguous responses optimize overall information transmission.

Figures (2)

• Figure 1: The information bottleneck problem for multiple variables. A signal $s$ carries information $I(s;x)$ about a relevant variable $x$, which is transmitted via a set of intermediate variables $C_i$. Each $C_i$ is controlled independently by $s$, conveying information $I(C_i;s)$, but they carry information about $x$ collectively, as measured by $I(\{C_i\};x)$
• Figure 2: Input/output relations $\bar{C}(s)$, their deficits $\rho$ from Eq (\ref{['rho_def']}), and consequences for information flow. (a) An invertible relation with $\rho = 0$. (b) A sawtooth relation that folds the dynamic range once, with $\rho = 1\,{\rm bit}$. (c) A sawtooth with two folds and $\rho = 2\,{\rm bits}$. (d) A nonlinear relation, still invertible and hence with $\rho =0$ as in (a). (e) A nonlinear relation with one symmetric fold and $\rho = 1\,{\rm bit}$ as in (b). (f) A double peaked relation, but since the minimum at $s=0.5$ is above the ones at $s=0, 1$ we find a fractional $\rho = 1.84\,{\rm bits}$. (g) The information plane for a system in which $s$ encodes the relevant variable $x$ linearly (inset) and information is shared equally between two channels $C_1$ and $C_2$, with $\bar{C}_1(s)$ being linear and $\bar{C}_2(s)$ being linear or piecewise linear. Theoretical predictions (colored lines), numerics (circles), and the optimum when there is a single channel with sufficient capacity (black line).