i-RevNet: Deep Invertible Networks

TL;DR

The paper questions the necessity of information loss in deep learning and introduces i-RevNet, an invertible CNN that preserves input information until the final class projection. It demonstrates that depth can induce contraction and linear separability without discarding information, achieving competitive ImageNet results with explicit input reconstruction via an inverse. The work shows that the inverse is well-behaved for reconstruction despite ill-conditioned local inversions and provides evidence of progressive contraction and a low-dimensional discriminative subspace. Altogether, it challenges the prevailing view that invariance requires information loss and offers a framework for analyzing depth-related representations through exact invertibility.

Abstract

It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

Figures (7)

• Figure 1: The main component of the $i$-RevNet and its inverse. RevNet blocks are interleaved with convolutional bottlenecks $\mathcal{F}_j$ and reshuffling operations $\mathcal{S}_j$ to ensure invertibility of the architecture and computational efficiency. The input is processed through a splitting operator $\tilde{\mathcal{S}}$, and output is merged through $\tilde{\mathcal{M}}$. Observe that the inverse network is obtained with minimal adaptations.
• Figure 2: Illustration of the invertible down-sampling
• Figure 3: Training loss of the $i$-RevNet (b), compared to the ResNet, on ImageNet.
• Figure 4: Normalized sorted singular values of $\partial \Phi_x$.
• Figure 5: This graphic displays several reconstructed sequences $\{x^t\}_t$. The left image corresponds to $x^0$ and the right image to $x^1$.