Self-Supervised Neural Architecture Search for Multimodal Deep Neural Networks

TL;DR

This work addresses the challenge of designing multimodal neural architectures without relying on large labeled datasets by introducing self-supervised neural architecture search (SSNAS) for multimodal DNNs. Building on gradient-based search and SimCLR-style contrastive learning, the method searches a fusion-structure space and pretrains representations entirely with unlabeled data, followed by a lightweight supervised fine-tuning stage. Experiments on MM-IMDB demonstrate that the proposed approach can yield architectures competitive with supervised NAS methods, particularly under scarce label regimes, highlighting the practical potential of unlabeled-data-driven multimodal NAS. Overall, the paper demonstrates a viable path to scalable multimodal NAS with reduced labeling requirements, enabling broader applicability of fusion models across tasks with limited annotations.

Abstract

Neural architecture search (NAS), which automates the architectural design process of deep neural networks (DNN), has attracted increasing attention. Multimodal DNNs that necessitate feature fusion from multiple modalities benefit from NAS due to their structural complexity; however, constructing an architecture for multimodal DNNs through NAS requires a substantial amount of labeled training data. Thus, this paper proposes a self-supervised learning (SSL) method for architecture search of multimodal DNNs. The proposed method applies SSL comprehensively for both the architecture search and model pretraining processes. Experimental results demonstrated that the proposed method successfully designed architectures for DNNs from unlabeled training data.

Figures (4)

• Figure 1: Network structure and contrastive learning processes of the proposed method.
• Figure 2: Algorithm of architecture search for multimodal DNNs using contrastive loss
• Figure 3: Comparison with the previous method.
• Figure 4: Obtained network structures.