On the Geometry of Deep Learning

Abstract

In this paper, we overview one promising avenue of progress at the mathematical foundation of deep learning: the connection between deep networks and function approximation by affine splines (continuous piecewise linear functions in multiple dimensions). In particular, we will overview work over the past decade on understanding certain geometrical properties of a deep network's affine spline mapping, in particular how it tessellates its input space. As we will see, the affine spline connection and geometrical viewpoint provide a powerful portal through which to view, analyze, and improve the inner workings of a deep network.

Figures (12)

• Figure 1: A 6-layer deep network. The purple, blue, and yellow nodes represent the input, neurons, and output, respectively, while the edges represent the affine transformation and activation effected by each layer. The width of layer 2 is 5, for example. The links between the nodes represent the elements of the weight matrices $\boldsymbol{W}^{(\ell)}$. The sum with the bias $\boldsymbol{b}^{(\ell)}$ and subsequent activation $\sigma(\cdot)$ are implicitly performed at each neuron.
• Figure 2: At left, a one-dimensional continuous piecewise linear function that we refer to as an affine spline. At right, the ReLU activation function (\ref{['eq:ReLU']}) at the heart of many of today's deep networks.
• Figure 3: Input space tessellation $\Omega$ of the two-dimensional input space (below) and affine spline mapping $f(\boldsymbol{x})$ (above) for a toy deep network of depth $L=4$ and width 20. Also depicted is a training data pair $(\boldsymbol{x}_i,y_i)$ and the prediction $\widehat{y_i}$.
• Figure 4: A deep network layer tessellates its input space into convex polytopal tiles via a hyperplane arrangement, with each hyperplane corresponding to one neuron at the output of the layer. In this two-dimensional example assuming ReLU activation, the red line indicates the one-dimensional hyperplane corresponding to the $k$-th neuron in the first layer.
• Figure 5: SplineCam visualization of a two-dimensional slice through the affine spline tessellation of the 4096-dimensional input space of a 5-layer ConvNet of average width 160 trained to classify $64 \times 64$ digital photos of cats. The stars denote the three training images that define the plane and the red lines the decision boundaries between the two classes. (Adapted from splinecam.)