Towards evolution of Deep Neural Networks through contrastive Self-Supervised learning

TL;DR

The paper addresses the design-time cost of deep neural networks and the reliance on labeled data by proposing EvoDeNSS, a framework that couples neuroevolution with self-supervised learning to evolve deep networks for image classification. It trains networks either under full supervision or via self-supervised pretraining using Barlow Twins, followed by a fixed linear classifier for the downstream task. On CIFAR-10, EvoDeNSS demonstrates that SSL-evolved networks can achieve competitive performance with limited labeled data and exhibit different structural patterns than fully supervised counterparts. The work contributes an integrated methodology, provides empirical results, and releases code to facilitate future exploration of adaptive SSL-driven evolution.

Abstract

Deep Neural Networks (DNNs) have been successfully applied to a wide range of problems. However, two main limitations are commonly pointed out. The first one is that they require long time to design. The other is that they heavily rely on labelled data, which can sometimes be costly and hard to obtain. In order to address the first problem, neuroevolution has been proved to be a plausible option to automate the design of DNNs. As for the second problem, self-supervised learning has been used to leverage unlabelled data to learn representations. Our goal is to study how neuroevolution can help self-supervised learning to bridge the gap to supervised learning in terms of performance. In this work, we propose a framework that is able to evolve deep neural networks using self-supervised learning. Our results on the CIFAR-10 dataset show that it is possible to evolve adequate neural networks while reducing the reliance on labelled data. Moreover, an analysis to the structure of the evolved networks suggests that the amount of labelled data fed to them has less effect on the structure of networks that learned via self-supervised learning, when compared to individuals that relied on supervised learning.

Figures (9)

• Figure 1: Overview of the process for an image classification problem.
• Figure 2: Overview of Fast-DENSER.
• Figure 3: Overview of individual evaluation under different learning paradigms.
• Figure 4: Barlow Twins algorithm
• Figure 5: Set splits created by the dataset partitioning process. Coloured sets represent the ones used during the evolutionary process by supervised and/or self-supervised learning, whereas the test set is used at the end to check the final performance.