Tighter Bounds on the Information Bottleneck with Application to Deep Learning

TL;DR

This work introduces a new and tighter variational bound for the Information Bottleneck, improving performance of previous IB-inspired DNNs and providing a simple method to significantly enhance the adversarial robustness of classifier DNNs.

Abstract

Deep Neural Nets (DNNs) learn latent representations induced by their downstream task, objective function, and other parameters. The quality of the learned representations impacts the DNN's generalization ability and the coherence of the emerging latent space. The Information Bottleneck (IB) provides a hypothetically optimal framework for data modeling, yet it is often intractable. Recent efforts combined DNNs with the IB by applying VAE-inspired variational methods to approximate bounds on mutual information, resulting in improved robustness to adversarial attacks. This work introduces a new and tighter variational bound for the IB, improving performance of previous IB-inspired DNNs. These advancements strengthen the case for the IB and its variational approximations as a data modeling framework, and provide a simple method to significantly enhance the adversarial robustness of classifier DNNs.

Figures (5)

• Figure 1: The information plane and curve: rate-distortion ratio over $\beta$. At $\beta=0$ the representation is compressed but uninformative (maximal compression), at $\beta\rightarrow\infty$ the representation is informative but potentially overfitted (maximal information). Adapted from Slonim2002.
• Figure 2: Information plane scatters of different DNN layers (colors) in 50 randomized networks. From ShwartzZiv2017. Left are initial weights, Right are at 400 epochs. Our study reproduced similar yet repetitive behavior on complicated high dimensional tasks, as elaborated in Section \ref{['subsec:Evaluation-and-analysis']} and in Figure \ref{['estimated_info_plane']}.
• Figure 3: Estimated information plane metrics per epoch for VUB trained on the IMDB dataset with $\beta=0.001$. $I(Z;X)$ is approximated by $H(R)-H(Z|X)$ and $\frac{1}{CE(Y;\hat{Y})}$ is used as an analog for $I(Z;Y)$. The epochs have been grouped and color-coded in intervals of 30 epochs in the order: Orange (0-30), gray (30-60), yellow (60-90), green (90-120) and red (120-150). We notice recurring patterns of distortion reduction followed by rate increase, resembling the ERM and representation compression stages described by ShwartzZiv2017.
• Figure 4: Successful targeted CW attack examples. Images are perturbations of previously successfully classified instances from the ImageNet validation set. The target label is 'Soccer ball'. Average $L_{2}$ distance required for a successful attack is shown on the left. The higher the required $L_{2}$ distance the greater the visible change required to fool the model. Original and wrongly assigned labels are listed at the top of each image. Mind the difference in noticeable change as compared to the FGS perturbations presented in Figure \ref{['fig:untargeted_examples']}, and between VIB and VUB perturbations.
• Figure 5: Successful untargeted FGS attack examples. Images are perturbations of previously successfully classified instances from the ImageNet validation set. Perturbation magnitude is determined by the parameter $\epsilon$ shown on the left, the higher the more perturbed. Original and wrongly assigned labels are listed at the top of each image. Notice the deterioration of image quality as $\epsilon$ increases.