Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck

TL;DR

The paper introduces the Generalized Symmetric Information Bottleneck (GSIB) for jointly compressing two high-dimensional variables and analyzes the data-efficiency of simultaneous versus independent compression. By deriving bounds on loss-function fluctuations via McDiarmid’s inequality and examining mean- and mean-squared errors, it shows that GSIB typically achieves smaller estimation bias and comparable variance, making simultaneous reduction more data-efficient than two separate GIBs in realistic settings. The authors also discuss the deterministic limit (DSIB), potential fixed-point issues, and provide appendices with detailed derivations of the GSIB updates and error calculations. Collectively, the work suggests a general principle: simultaneous dimensionality reduction can require substantially less data to achieve the same accuracy, with implications for physics-inspired modeling, neuroscience, and systems biology.

Abstract

The Symmetric Information Bottleneck (SIB), an extension of the more familiar Information Bottleneck, is a dimensionality reduction technique that simultaneously compresses two random variables to preserve information between their compressed versions. We introduce the Generalized Symmetric Information Bottleneck (GSIB), which explores different functional forms of the cost of such simultaneous reduction. We then explore the dataset size requirements of such simultaneous compression. We do this by deriving bounds and root-mean-squared estimates of statistical fluctuations of the involved loss functions. We show that, in typical situations, the simultaneous GSIB compression requires qualitatively less data to achieve the same errors compared to compressing variables one at a time. We suggest that this is an example of a more general principle that simultaneous compression is more data efficient than independent compression of each of the input variables.