LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Search

TL;DR

The paper addresses HW-NAS by combining LLM-based design with explicit diversity mechanisms to overcome exploration bias and prohibitively high training costs in traditional supernet-based NAS.It introduces LLM-NAS, which partitions the search space by architectural complexity, employs a co-evolving knowledge base to guide LLM-driven mutations and crossovers, and uses a training-free evaluator to rapidly score candidates.Empirical results on HW-NAS-Bench show that LLM-NAS achieves a more complete and dominant Pareto front with higher Hypervolume and lower IGD, while reducing search time from GPU-days to minutes.The approach extends to ViT search spaces, demonstrating generalizability, and ablation studies confirm the critical roles of partitioning, the LLM operator, and the zero-cost predictor in performance gains.

Abstract

Hardware-Aware Neural Architecture Search (HW-NAS) requires joint optimization of accuracy and latency under device constraints. Traditional supernet-based methods require multiple GPU days per dataset. Large Language Model (LLM)-driven approaches avoid training a large supernet and can provide quick feedback, but we observe an exploration bias: the LLM repeatedly proposes neural network designs within limited search space and fails to discover architectures across different latency ranges in the entire search space. To address this issue, we propose LLM-NAS: an LLM-driven Neural Architecture Search that can generate neural networks with high accuracy and low latency with reduced search cost. Our proposed LLM-NAS has three key components: 1) a complexity-driven partitioning engine that divides the search space by complexity to enforce diversity and mitigate exploration bias; 2) an LLM-powered architecture prompt co-evolution operator, in which the LLM first updates a knowledge base of design heuristics based on results from the previous round, then performs a guided evolution algorithm on architectures with prompts that incorporate this knowledge base. Prompts and designs improve together across rounds which avoids random guesswork and improve efficiency; 3) a zero-cost predictor to avoid training a large number of candidates from scratch. Experimental results show that on HW-NAS-Bench, LLM-NAS can achieve overall higher HV, lower IGD, and up to 54% lower latency than baselines at similar accuracy. Meanwhile, the search cost drops from days to minutes compared with traditional supernet baselines.

Figures (9)

• Figure 1: Comparison of three generation strategies on HW-NAS-Bench (Edge GPU, CIFAR-10): normal prompt (orange), latency-optimized prompt (blue), and LLM-NAS (green). Latency-optimized prompting increases coverage compared to standard prompting but still leaves gaps, while LLM-NAS achieves near-complete coverage across latency ranges
• Figure 2: The LLM-NAS framework. The search space is partitioned into complexity-based niches, where an LLM performs parallel evolutionary search. The individual results are then aggregated to form the final, complete Pareto front, mitigating exploration bias
• Figure 3: Analysis of the HW-NAS-Bench search space. The distribution of total parameters exhibits clear clustering
• Figure 4: The Co-evolve Prompt Generator in LLM-NAS. The LLM first acts as a reasoning engine, updating a Knowledge Base by analyzing prior results. This learned knowledge then informs the LLM's second role as an expert architect, where it generates new, rationale-driven architectures under specific constraints, creating a self-optimizing search process
• Figure 5: The Pareto front discovered by LLM-NAS for three AutoFormer search spaces on ImageNet. Latency is evaluated using a single NVIDIA A6000 GPU, and accuracy is estimated via a predictor