Learning to Compile Programs to Neural Networks

TL;DR

The paper tackles the problem of efficiently creating neural surrogates that mimic program behavior without resorting to costly per-program data collection or oversized universal models. It introduces neural surrogate compilation, using a Transformer-derived hypernetwork (CompNet) to generate initial surrogate networks directly from program text, thereby decoupling surrogate creation from execution. Key contributions include the CompNet architecture, the ExeStack and ExeStackCPN datasets, and evaluations on ExeStackCPN and ParrotBenchCPN that demonstrate substantial gains in data efficiency ($1.9$-$9.5\times$ lower error), fidelity ($1.0$-$1.3\times$ higher similarity), and training speed ($4.3$-$7.3\times$ fewer epochs) relative to training from scratch. This approach offers a practical path to deploying efficient, transferable neural surrogates in resource-constrained settings and provides a scalable framework for compiling program text into neural representations.

Abstract

A $\textit{neural surrogate of a program}$ is a neural network that mimics the behavior of a program. Researchers have used these neural surrogates to automatically tune program inputs, adapt programs to new settings, and accelerate computations. Researchers traditionally develop neural surrogates by training on input-output examples from a single program. Alternatively, language models trained on a large dataset including many programs can consume program text, to act as a neural surrogate. Using a language model to both generate a surrogate and act as a surrogate, however, leading to a trade-off between resource consumption and accuracy. We present $\textit{neural surrogate compilation}$, a technique for producing neural surrogates directly from program text without coupling neural surrogate generation and execution. We implement neural surrogate compilers using hypernetworks trained on a dataset of C programs and find that they produce neural surrogates that are $1.9$-$9.5\times$ as data-efficient, produce visual results that are $1.0$-$1.3\times$ more similar to ground truth, and train in $4.3$-$7.3\times$ fewer epochs than neural surrogates trained from scratch.