On Diversity in Discriminative Neural Networks

TL;DR

The paper addresses the lack of deliberate diversity mechanisms in discriminative neural networks and explores how telecom-style diversity concepts can improve self- and semi-supervised learning. It proposes a two-part architecture with an encoder and an MLP using competition-based sparse layers and data augmentation-inspired redundancy, leveraging both unlabeled data and a small labeled set. Key results show a record self-supervised MNIST accuracy around 99.5% and CIFAR-10 semi-supervised accuracy of about 94% with 25 labeled samples per class, using sparsity to gain efficiency. The work demonstrates the viability of diversity-inspired designs to achieve high performance with limited labels and points to hyperparameter sensitivity and future improvements.

Abstract

Diversity is a concept of prime importance in almost all disciplines based on information processing. In telecommunications, for example, spatial, temporal, and frequency diversity, as well as redundant coding, are fundamental concepts that have enabled the design of extremely efficient systems. In machine learning, in particular with neural networks, diversity is not always a concept that is emphasized or at least clearly identified. This paper proposes a neural network architecture that builds upon various diversity principles, some of them already known, others more original. Our architecture obtains remarkable results, with a record self-supervised learning accuracy of 99. 57% in MNIST, and a top tier promising semi-supervised learning accuracy of 94.21% in CIFAR-10 using only 25 labels per class.

Figures (3)

• Figure 1: Overview of the proposed learning method for unlabeled data. For MNIST, the size of the blocks is 8 and 10 for CIFAR-10.
• Figure 2: Proposed network architecture for MNIST
• Figure 3: Proposed network architecture for CIFAR-10