Multi-Task Neural Architecture Search Using Architecture Embedding and Transfer Rank

TL;DR

Ranking disorder in cross-task neural architecture search (NAS) undermines transfer performance across tasks. KTNAS integrates architecture embedding via graph-based embeddings and transfer-rank guided selection within an evolutionary multi-task NAS framework to promote positive transfer while reducing negative transfer. It formalizes multi-task NAS across $N$ tasks with objective $\oldmath{\\alpha_i^*}=\\arg\\min_{\\alpha_i} L(\\omega_i(\\alpha_i),\\alpha_i; D^i_{val})$, and uses a node2vec-based embedding to predict performance and guide cross-task mating. Extensive experiments on NASBench-201, TransNAS-Bench-101, and DARTs demonstrate improved search efficiency and downstream task accuracy versus baselines, with ablations confirming the critical role of transfer rank in enabling effective transfer.

Abstract

Multi-task neural architecture search (NAS) enables transferring architectural knowledge among different tasks. However, ranking disorder between the source task and the target task degrades the architecture performance on the downstream task. We propose KTNAS, an evolutionary cross-task NAS algorithm, to enhance transfer efficiency. Our data-agnostic method converts neural architectures into graphs and uses architecture embedding vectors for the subsequent architecture performance prediction. The concept of transfer rank, an instance-based classifier, is introduced into KTNAS to address the performance degradation issue. We verify the search efficiency on NASBench-201 and transferability to various vision tasks on Micro TransNAS-Bench-101. The scalability of our method is demonstrated on DARTs search space including CIFAR-10/100, MNIST/Fashion-MNIST, MedMNIST. Experimental results show that KTNAS outperforms peer multi-task NAS algorithms in search efficiency and downstream task performance. Ablation studies demonstrate the vital importance of transfer rank for transfer performance.

Figures (6)

• Figure 1: Different knowledge transfer processes between (a) model-based transfer learning, (b) EMT-NAS and (c) KTNAS. Best viewed in color.
• Figure 2: Ranking correlation of transferred architectures among 7 tasks on Micro TransNAS-Bench-101.
• Figure 3: The flow chart of KTNAS for two-task scenarios. Green denotes self-evolution. Blue denotes cross-task knowledge transfer. Best viewed in color.
• Figure 4: An example for transfer rank calculation.
• Figure 5: The average convergence curves on NAS201. The shaded area denotes standard variance over 10 runs.