Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Networks Learn Distance Metrics

Alan Oursland

TL;DR

This paper investigates whether neural networks inherently learn distance-based representations rather than relying solely on activation magnitudes. It formulates a geometric-statistical framework centered on distance metrics, notably the Mahalanobis distance, and tests six MNIST architectures forced to adopt either distance- or intensity-based representations. The authors introduce OffsetL2, a prototype-distance architecture, and show through extensive experiments that representation type and architectural details critically shape performance, with OffsetL2 achieving high accuracy and remarkable stability. The work argues for explicitly modeling distances to learned prototypes as a principled design choice, with potential to improve interpretability and guide future architecture development in deep learning.

Abstract

Neural networks may naturally favor distance-based representations, where smaller activations indicate closer proximity to learned prototypes. This contrasts with intensity-based approaches, which rely on activation magnitudes. To test this hypothesis, we conducted experiments with six MNIST architectural variants constrained to learn either distance or intensity representations. Our results reveal that the underlying representation affects model performance. We develop a novel geometric framework that explains these findings and introduce OffsetL2, a new architecture based on Mahalanobis distance equations, to further validate this framework. This work highlights the importance of considering distance-based learning in neural network design.

Neural Networks Learn Distance Metrics

TL;DR

This paper investigates whether neural networks inherently learn distance-based representations rather than relying solely on activation magnitudes. It formulates a geometric-statistical framework centered on distance metrics, notably the Mahalanobis distance, and tests six MNIST architectures forced to adopt either distance- or intensity-based representations. The authors introduce OffsetL2, a prototype-distance architecture, and show through extensive experiments that representation type and architectural details critically shape performance, with OffsetL2 achieving high accuracy and remarkable stability. The work argues for explicitly modeling distances to learned prototypes as a principled design choice, with potential to improve interpretability and guide future architecture development in deep learning.

Abstract

Neural networks may naturally favor distance-based representations, where smaller activations indicate closer proximity to learned prototypes. This contrasts with intensity-based approaches, which rely on activation magnitudes. To test this hypothesis, we conducted experiments with six MNIST architectural variants constrained to learn either distance or intensity representations. Our results reveal that the underlying representation affects model performance. We develop a novel geometric framework that explains these findings and introduce OffsetL2, a new architecture based on Mahalanobis distance equations, to further validate this framework. This work highlights the importance of considering distance-based learning in neural network design.

Paper Structure

This paper contains 24 sections, 4 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Features, Representations, and Distance Metrics
  5. Distance vs. Intensity Representations
  6. Theoretical Foundations: Mahalanobis Distance
  7. Geometric Interpretations of Representations
  8. Activation Functions and Representational Preferences
  9. Experimental Design
  10. Objectives
  11. Model Design
  12. Experimental Setup
  13. Experimental Results
  14. Baseline Performance
  15. Intensity Learning Models
  16. ...and 9 more sections

Figures (1)

  • Figure 1: Class distributions in the latent space show overlapping clusters with varying statistical properties. Each class exhibits distinct characteristics (mean, variance, skewness), with overlap patterns varying across different linear projections - typical behavior for non-linearly separable data.