Design Principle Transfer in Neural Architecture Search via Large Language Models

TL;DR

The paper tackles the inefficiency of multi-task NAS by introducing design principle transfer, where a pre-trained LLM learns general design principles from established architectures and constrains the search space for new tasks. The LAPT framework then adaptively refines these principles to task-specific subspaces, decoupling knowledge transfer from the NAS method. Empirical results across NAS201, Trans101, and DARTs show that searching within the refined spaces yields state-of-the-art or competitive performance with substantial speedups, validating the approach. This work enhances NAS practicality and interpretability by automating principle extraction and space refinement, paving the way for broader cross-task applicability of NAS.

Abstract

Transferable neural architecture search (TNAS) has been introduced to design efficient neural architectures for multiple tasks, to enhance the practical applicability of NAS in real-world scenarios. In TNAS, architectural knowledge accumulated in previous search processes is reused to warm up the architecture search for new tasks. However, existing TNAS methods still search in an extensive search space, necessitating the evaluation of numerous architectures. To overcome this challenge, this work proposes a novel transfer paradigm, i.e., design principle transfer. In this work, the linguistic description of various structural components' effects on architectural performance is termed design principles. They are learned from established architectures and then can be reused to reduce the search space by discarding unpromising architectures. Searching in the refined search space can boost both the search performance and efficiency for new NAS tasks. To this end, a large language model (LLM)-assisted design principle transfer (LAPT) framework is devised. In LAPT, LLM is applied to automatically reason the design principles from a set of given architectures, and then a principle adaptation method is applied to refine these principles progressively based on the new search results. Experimental results show that LAPT can beat the state-of-the-art TNAS methods on most tasks and achieve comparable performance on others.

Figures (10)

• Figure 1: Overview of the proposed LAPT. This framework consists of two stages. In the learning stage of design principles, LLM is driven by specific prompts to learn general design principles from a set of architectures. In the architecture search stage, the learned principles are applied to initialize the search space for each new task. Then, architectures found in the refined search space are used to update these principles, aiming to build the task-specific search space.
• Figure 2: The Python code-based prompt can help LLMs gain awareness of the right CNN architecture from its architectural parameters which are represented as "Layer_list".
• Figure 3: Design principles separately learned by LAPT and experts. Our LLM-based framework not only learns the identical principles (i.e., $P1-P3$) with experts but also can reason other valuable principles $P4$ and $P5$.
• Figure 4: Model rank of different LAPT versions on Trans101.
• Figure 5: Prompts used to drive LLMs to learn and adapt the design principles from architectures in NAS201.