RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design

TL;DR

RevoNAD tackles neural architecture design by bridging LLM-driven reasoning with feedback-aligned evolutionary search. It combines multi-round multi-expert consensus (MMC), adaptive reflective exploration (ARDE), and Pareto-guided evolutionary selection (PES) to produce diverse, robust architectures. The approach achieves state-of-the-art performance on CIFAR-10/100, ImageNet16-120, COCO-5K, and Cityscapes, with validated ablations and transfer studies. The work highlights the value of structured reasoning and adaptive exploration for scalable, interpretable NAD.

Abstract

Recent progress in leveraging large language models (LLMs) has enabled Neural Architecture Design (NAD) systems to generate new architecture not limited from manually predefined search space. Nevertheless, LLM-driven generation remains challenging: the token-level design loop is discrete and non-differentiable, preventing feedback from smoothly guiding architectural improvement. These methods, in turn, commonly suffer from mode collapse into redundant structures or drift toward infeasible designs when constructive reasoning is not well grounded. We introduce RevoNAD, a reflective evolutionary orchestrator that effectively bridges LLM-based reasoning with feedback-aligned architectural search. First, RevoNAD presents a Multi-round Multi-expert Consensus to transfer isolated design rules into meaningful architectural clues. Then, Adaptive Reflective Exploration adjusts the degree of exploration leveraging reward variance; it explores when feedback is uncertain and refines when stability is reached. Finally, Pareto-guided Evolutionary Selection effectively promotes architectures that jointly optimize accuracy, efficiency, latency, confidence, and structural diversity. Across CIFAR10, CIFAR100, ImageNet16-120, COCO-5K, and Cityscape, RevoNAD achieves state-of-the-art performance. Ablation and transfer studies further validate the effectiveness of RevoNAD in allowing practically reliable, and deployable neural architecture design.

Figures (8)

• Figure 1: Comparison of search efficiency on CIFAR10. RevoNAD finds stronger architectures with fewer trials, showing substantially improved design efficiency compared to prior NAD/NAS methods.
• Figure 2: RevoNAD Orchestrator. Starting from a base model, a multi-expert ideation module distills literature into architectural inspirations, which are then fed to an LLM-based reflective design exploration module that proposes new candidate architectures. The candidates are trained and evaluated, and a Pareto-guided evolution preserves diverse high-quality models while updating survivors for the next generation.
• Figure 3: Multi-round Multi-expert Consensus refinement across varying numbers of experts and refinement rounds. R1–E2 denotes 1 round and 2 experts, respectively. Note that the red dotted box highlights the most significant improvement.
• Figure 4: Adaptive exploration schedule. High joint dynamics of $\varepsilon_n$ encourages exploration, while low variance drives exploitation.
• Figure 5: Cross-dataset direct transfer under different confidence weights $\rho_c$. Each heatmap reports the test accuracy, with colors indicating the accuracy deviation, when architectures searched on a source dataset (rows) are directly transferred and evaluated on a target dataset (columns). Diagonal entries denotes Oracle in-domain performance where the source and target datasets are the same.