A Transformer-based Neural Architecture Search Method

TL;DR

The paper addresses neural architecture search for Transformer-based machine translation by jointly optimizing translation quality and perplexity. It introduces MO-Trans, a MOEA/D-based NAS framework that searches cross multihead attention patterns and varying encoder/decoder block configurations, using BLEU and perplexity as dual objectives. The authors show that incorporating perplexity as an auxiliary metric helps discover architectures with superior BLEU scores compared to a baseline Transformer, across English→German and German→English on Multi30k. This approach advances NAS for MT by demonstrating effective multi-objective optimization and principled cross-attention design choices, with practical impact on constructing more accurate translation models.

Abstract

This paper presents a neural architecture search method based on Transformer architecture, searching cross multihead attention computation ways for different number of encoder and decoder combinations. In order to search for neural network structures with better translation results, we considered perplexity as an auxiliary evaluation metric for the algorithm in addition to BLEU scores and iteratively improved each individual neural network within the population by a multi-objective genetic algorithm. Experimental results show that the neural network structures searched by the algorithm outperform all the baseline models, and that the introduction of the auxiliary evaluation metric can find better models than considering only the BLEU score as an evaluation metric.

Figures (5)

• Figure 1: Searching space of combinations of blocks
• Figure 2: Cases in different numbers of encoder blocks and decoder blocks
• Figure 3: Illustration of crossover between individuals
• Figure 4: Illustration of the Pareto front of the population when the MO-Trans algorithm finished running
• Figure 5: Schematic of the best model searched by the MO-Trans algorithm