Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Design

TL;DR

The paper investigates how to effectively use large language models to generate computer vision neural network architectures, addressing two practical bottlenecks: prompt design and code deduplication. It introduces Few-Shot Architecture Prompting (FSAP) to determine the optimal number of supporting examples (found to be $n=3$) and a whitespace-normalized MD5-based hash validation to prevent redundant training, achieving about 100x speedups over AST-based checks. Through large-scale experiments spanning seven CV benchmarks and 1,900 architectures, the study demonstrates dataset-balanced evaluation as essential to fair cross-task comparisons and finds that FSAP with $n=3$ yields the best trade-off between diversity and context, with significant gains on complex tasks like CIFAR-100. The approach markedly improves scalability and reliability of LLM-driven CV architecture design, saving substantial compute while providing actionable guidelines for researchers with limited resources. Limitations include single-epoch training and potential semantic dedup beyond formatting, suggesting future work in multi-task CV settings and explainable architecture synthesis.

Abstract

Automated neural network architecture design remains a significant challenge in computer vision. Task diversity and computational constraints require both effective architectures and efficient search methods. Large Language Models (LLMs) present a promising alternative to computationally intensive Neural Architecture Search (NAS), but their application to architecture generation in computer vision has not been systematically studied, particularly regarding prompt engineering and validation strategies. Building on the task-agnostic NNGPT/LEMUR framework, this work introduces and validates two key contributions for computer vision. First, we present Few-Shot Architecture Prompting (FSAP), the first systematic study of the number of supporting examples (n = 1, 2, 3, 4, 5, 6) for LLM-based architecture generation. We find that using n = 3 examples best balances architectural diversity and context focus for vision tasks. Second, we introduce Whitespace-Normalized Hash Validation, a lightweight deduplication method (less than 1 ms) that provides a 100x speedup over AST parsing and prevents redundant training of duplicate computer vision architectures. In large-scale experiments across seven computer vision benchmarks (MNIST, CIFAR-10, CIFAR-100, CelebA, ImageNette, SVHN, Places365), we generated 1,900 unique architectures. We also introduce a dataset-balanced evaluation methodology to address the challenge of comparing architectures across heterogeneous vision tasks. These contributions provide actionable guidelines for LLM-based architecture search in computer vision and establish rigorous evaluation practices, making automated design more accessible to researchers with limited computational resources.

Figures (2)

• Figure 1: Complete architecture generation pipeline showing integration of Few-Shot Architecture Prompting (FSAP) and Whitespace-Normalized Hash Validation into the NNGPT system. Our contributions (highlighted) enhance both generation quality and computational efficiency.
• Figure 2: Context overflow phenomenon. Balanced mean accuracy (left axis) peaks at n=3 (53.1%), while generation success (right axis) drops exponentially beyond n=3, with catastrophic failure at n=6 (only 7 models generated vs. 1,268 at n=1).