From Memorization to Creativity: LLM as a Designer of Novel Neural-Architectures

TL;DR

This work investigates whether a code-capable LLM can progressively become a robust designer of neural architectures by learning from its own successful generations in a closed-loop synthesis loop. By combining low-cost, single-epoch CIFAR-10 evaluations with a MinHash-Jaccard novelty filter and LoRA-based iterative fine-tuning, the study shows the generator improves in validity and early-learning performance while maintaining architectural diversity over 22 cycles. Key findings include a rise in mean first-epoch accuracy from 28.06% to 50.99%, a peak valid-generation rate of 74.5%, and the discovery of 455 novel architectures, illustrating that LLMs can internalize empirical, non-textual rewards to shape architectural priors. The results offer a scalable blueprint for turning stochastic code generators into autonomous, performance-driven NAS components, with implications for integrating such priors into broader NAS frameworks and multi-task, multi-objective contexts.

Abstract

Large language models (LLMs) excel in program synthesis, yet their ability to autonomously navigate neural architecture design--balancing syntactic reliability, performance, and structural novelty--remains underexplored. We address this by placing a code-oriented LLM within a closed-loop synthesis framework, analyzing its evolution over 22 supervised fine-tuning cycles. The model synthesizes PyTorch convolutional networks which are validated, evaluated via low-fidelity performance signals (single-epoch accuracy), and filtered using a MinHash-Jaccard criterion to prevent structural redundancy. High-performing, novel architectures are converted into prompt-code pairs for iterative fine-tuning via parameter-efficient LoRA adaptation, initialized from the LEMUR dataset. Across cycles, the LLM internalizes empirical architectural priors, becoming a robust generator. The valid generation rate stabilizes at 50.6 percent (peaking at 74.5 percent), while mean first-epoch accuracy rises from 28.06 percent to 50.99 percent, and the fraction of candidates exceeding 40 percent accuracy grows from 2.04 percent to 96.81 percent. Analyses confirm the model moves beyond replicating existing motifs, synthesizing 455 high-performing architectures absent from the original corpus. By grounding code synthesis in execution feedback, this work provides a scalable blueprint for transforming stochastic generators into autonomous, performance-driven neural designers, establishing that LLMs can internalize empirical, non-textual rewards to transcend their training data.

Figures (5)

• Figure 1: Overview of the iterative architecture-synthesis loop: the LLM generates architectures, candidates are evaluated and filtered (validity, first-epoch accuracy, novelty), and the LLM is fine-tuned on selected outputs.
• Figure 2: Valid generation rate per cycle (1--22) with Wilson 95% confidence intervals.
• Figure 3: Overall analysis of the 22 fine-tuning cycles: (top-left) first-epoch accuracy trends; (top-right) quality distribution by accuracy threshold; (bottom-left) model selection and novelty; (bottom-right) training-data growth.
• Figure 4: Best, mean, and median first-epoch accuracy per cycle on CIFAR--10 (see Section \ref{['sec:results']}).
• Figure 5: Proportion of models with first-epoch accuracy $\geq 40\%$ per cycle (see Section \ref{['sec:results']}).