Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equation

TL;DR

This work addresses learning-curve extrapolation by incorporating neural network architecture into a continuous-time dynamical model. It introduces LC-GODE, which combines a graph-based architecture encoder with a latent ODE in a seq2seq variational framework to predict full learning curves from early epochs while providing uncertainty estimates. The approach improves extrapolation accuracy for both MLP and CNN curves and enhances NAS configuration ranking, achieving substantial speedups over full training cycles. By linking architecture topology to the learning dynamics, LC-GODE offers a scalable, architecture-aware tool for faster AutoML and more robust model selection.

Abstract

Learning curve extrapolation predicts neural network performance from early training epochs and has been applied to accelerate AutoML, facilitating hyperparameter tuning and neural architecture search. However, existing methods typically model the evolution of learning curves in isolation, neglecting the impact of neural network (NN) architectures, which influence the loss landscape and learning trajectories. In this work, we explore whether incorporating neural network architecture improves learning curve modeling and how to effectively integrate this architectural information. Motivated by the dynamical system view of optimization, we propose a novel architecture-aware neural differential equation model to forecast learning curves continuously. We empirically demonstrate its ability to capture the general trend of fluctuating learning curves while quantifying uncertainty through variational parameters. Our model outperforms current state-of-the-art learning curve extrapolation methods and pure time-series modeling approaches for both MLP and CNN-based learning curves. Additionally, we explore the applicability of our method in Neural Architecture Search scenarios, such as training configuration ranking.

Figures (6)

• Figure 1: Overall framework. (a) Given fixed training data and a specific task, each architecture defines a unique loss landscape. We employ a numerical optimization method (e.g., gradient descent) to sample a loss trajectory across this landscape. Our dataset consists of various architectures paired with their corresponding loss curves. The model input is an observation window, which it utilizes to predict the trajectory within the subsequent prediction window. (b) Our model architecture incorporates a graph encoder that captures the architectural structure by extracting a single embedding through message passing and global pooling. (c) We initialize a latent distribution at the first epoch. A GRU unit processes this information, generating a hidden vector at each observed timestamp based on the prior hidden state and the current loss value. (d) Finally, we integrate the ODE that governs the evolution of the latent loss states, with each time step modulated by the graph embedding.
• Figure 2: (a) Graph embedding projections. (b) Pairwise initial latent embedding distance vs performance difference.
• Figure 3: Extrapolation error (MAPE) for learning curves w.r.t. prediction length.
• Figure 4: True vs predicted best test accuracy for car, segment, CIFAR-10, CIFAR-100, respectively.
• Figure 5: Ablation study for encoders.