Metric as Transform: Exploring beyond Affine Transform for Interpretable Neural Network

TL;DR

It is found that metrics as transform performs similarly to affine transform when used in MultiLayer Perceptron or Convolutional Neural Network and an interpretable local dictionary based Neural Networks is developed and used to understand and reject adversarial examples.

Abstract

Artificial Neural Networks of varying architectures are generally paired with affine transformation at the core. However, we find dot product neurons with global influence less interpretable as compared to local influence of euclidean distance (as used in Radial Basis Function Network). In this work, we explore the generalization of dot product neurons to $l^p$-norm, metrics, and beyond. We find that metrics as transform performs similarly to affine transform when used in MultiLayer Perceptron or Convolutional Neural Network. Moreover, we explore various properties of Metrics, compare it with Affine, and present multiple cases where metrics seem to provide better interpretability. We develop an interpretable local dictionary based Neural Networks and use it to understand and reject adversarial examples.

Figures (20)

• Figure 1: Neuron Interpretation: (LEFT) 2D ReLU neuron at $\boldsymbol{w.x}+b \ge 0$, (RIGHT) 2D radial neuron at $\exp{(-\|\boldsymbol{x-w}\|^2)}$. The lines or circles show region of neuron firing, ($\times$) showing a data point and thickness of the lines indicating its activation magnitude. Zoom in for details.
• Figure 2: Dot product, cosine angle and distance between patches of image when the values are interpolated between black-gaussian-white. The images just above the scatter-points represents the interpolated image. The star ($\star$) symbol represents the reference point to which every interpolation is compared. Zoom in for details.
• Figure 3: Voronoi Diagram of (TOP) Linear and (BOT) Distance for LEFT: without using bias, MID: using bias, and RIGHT: using bias and shifting the center/weights by $[-0.5, -0.5]$
• Figure 4: We compare bounded functions with their unbounded counterparts (LEFT) Uniform functions (MID) convex function and (RIGHT) invex function. For measure of distance, we are concerned with bounded functions (BOT) with minimum at a point. Here, we can use function with bounded contour sets as measure of distance from the minima ($x^*$). To use the function itself as norm, we need to shift the minima to origin and the output of origin to zero.
• Figure 5: Mapping of double hellix using different measure of distance.