Data Complexity-aware Deep Model Performance Forecasting

TL;DR

This work addresses the high cost of selecting architectures and hyperparameters for deep learning by forecasting model performance before training. It introduces a two-stage framework that first estimates a dataset-intrinsic baseline accuracy using Data Complexity Measures (DCMs) and PCA, then predicts an architecture- and hyperparameter-induced offset with an XGBoost model conditioned on the baseline and architecture descriptors. The approach demonstrates strong predictive performance and generalization across multiple image datasets and domains, while offering practical guidelines for architecture depth based on a simple metric (Variance Mean) and a data-quality diagnostic signal (PC6). It advances predictive MLOps by enabling low-cost, interpretable pre-training decisions and data preprocessing recommendations, reducing reliance on costly trial-and-error. Limitations include focus on CNNs for vision and reliance on handcrafted DCMs, with future work extending to diverse architectures, tasks, and learned complexity representations.

Abstract

Deep learning models are widely used across computer vision and other domains. When working on the model induction, selecting the right architecture for a given dataset often relies on repetitive trial-and-error procedures. This procedure is time-consuming, resource-intensive, and difficult to automate. While previous work has explored performance prediction using partial training or complex simulations, these methods often require significant computational overhead or lack generalizability. In this work, we propose an alternative approach: a lightweight, two-stage framework that can estimate model performance before training given the understanding of the dataset and the focused deep model structures. The first stage predicts a baseline based on the analysis of some measurable properties of the dataset, while the second stage adjusts the estimation with additional information on the model's architectural and hyperparameter details. The setup allows the framework to generalize across datasets and model types. Moreover, we find that some of the underlying features used for prediction - such as dataset variance - can offer practical guidance for model selection, and can serve as early indicators of data quality. As a result, the framework can be used not only to forecast model performance, but also to guide architecture choices, inform necessary preprocessing procedures, and detect potentially problematic datasets before training begins.

Figures (12)

• Figure 1: Two-stage prediction framework.
• Figure 2: Distribution of model accuracy across datasets, highlighting the influence of dataset complexity.
• Figure 3: Explained variance ratio as a function of the number of principal components.
• Figure 4: Regression performance vs. number of PCA components. Metrics include $R^2$, MSE, and MAE. The best performance is achieved at $N = 7$.
• Figure 5: Loadings of DCMs on PC1: Local non-linear entanglement.