Neural Networks Use Distance Metrics

TL;DR

These findings challenge the prevailing intensity-based interpretation of neural network activations and offer new insights into their learning and decision-making processes.

Abstract

We present empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations. We independently manipulate both distance and intensity properties of internal activations in trained models, finding that both architectures are highly sensitive to small distance-based perturbations while maintaining robust performance under large intensity-based perturbations. These findings challenge the prevailing intensity-based interpretation of neural network activations and offer new insights into their learning and decision-making processes.

Figures (3)

• Figure 1: Effects of intensity scaling and distance offset perturbations on model accuracy. Shaded regions represent 95% confidence intervals across 20 runs.
• Figure 2: This series of figures illustrates how linear nodes process features using ReLU and Absolute Value activation functions. Each blue peak represents a feature (a-e), with the red dashed line showing the decision boundary. The top row shows features after linear projection but before activation. The bottom row shows how ReLU and Absolute Value functions transform these projections, highlighting their distinct effects on feature space.
• Figure 3: Effects of decision boundary offsets on feature representation. Negative offsets (top row) and positive offsets (bottom row) demonstrate how shifting the decision boundary affects feature selection in ReLU and Abs activated nodes.