Comparison of Embedded Spaces for Deep Learning Classification

TL;DR

The paper addresses the need for well-structured embedded spaces in deep classification. It surveys and compares embedding-design methods, including softmax, angular-margin, center, contrastive, triplet, and regression losses, and discusses normalization of weights and features. Through qualitative 2D and 3D visualizations on MNIST, Fashion-MNIST, and CIFAR-10, it demonstrates how different losses shape embedding geometry and interpretability. The results offer practical guidance for improving open-set recognition, few-shot learning, and explainability by deliberately shaping the embedding space prior to the final classifier.

Abstract

Embedded spaces are a key feature in deep learning. Good embedded spaces represent the data well to support classification and advanced techniques such as open-set recognition, few-short learning and explainability. This paper presents a compact overview of different techniques to design embedded spaces for classification. It compares different loss functions and constraints on the network parameters with respect to the achievable geometric structure of the embedded space. The techniques are demonstrated with two and three-dimensional embeddings for the MNIST, Fashion MNIST and CIFAR-10 datasets, allowing visual inspection of the embedded spaces.

Figures (7)

• Figure 1: Typical convolutional network for classification with the embedded space as it is considered in this paper
• Figure 2: Comparison of different activation functions (linear, sigmoid, relu, selu Geron.October2022) of the penultimate layer, i.e. right before the embedding (loss here is softmax loss). The linear activation provides the most suitable structure. (MNIST dataset)
• Figure 3: Embedded space for MNIST with ordinary softmax loss (left) and softmax loss with normalized weights for a more regular geometry (right)
• Figure 4: Angular margin loss (cosface) for different margins for $m=0,0.05$ and $0.2$. The right column shows the projection of the features to the unit circle. (MNIST)
• Figure 5: Center loss for an MNIST embedding without (left) and with (right) weight normalization. The classes are more compact than with ordinary softmax loss (Figure \ref{['fig:softmax']})