Anytime Neural Architecture Search on Tabular Data

TL;DR

This paper tackles the need for anytime neural architecture search on tabular data by introducing ATLAS, a two-phase framework that blends a fast training-free filtering stage (via the tabular-specific ExpressFlow proxy) with a training-based refinement stage scheduled by a budget-aware coordinator. It introduces NAS-Bench-Tabular, a comprehensive benchmark of about 160{,}000 architectures across three real-world datasets, and demonstrates ExpressFlow's strong correlation with actual performance, enabling rapid yet informative exploration. Empirical results show ATLAS achieves up to $82.75\times$ speedups over training-based NAS baselines and delivers better architectures as more budget becomes available, effectively enabling anytime NAS for tabular tasks. The approach advances NAS benchmarking for tabular data and provides practical AutoML tooling for latency-constrained deployment.

Abstract

The increasing demand for tabular data analysis calls for transitioning from manual architecture design to Neural Architecture Search (NAS). This transition demands an efficient and responsive anytime NAS approach that is capable of returning current optimal architectures within any given time budget while progressively enhancing architecture quality with increased budget allocation. However, the area of research on Anytime NAS for tabular data remains unexplored. To this end, we introduce ATLAS, the first anytime NAS approach tailored for tabular data. ATLAS introduces a novel two-phase filtering-and-refinement optimization scheme with joint optimization, combining the strengths of both paradigms of training-free and training-based architecture evaluation. Specifically, in the filtering phase, ATLAS employs a new zero-cost proxy specifically designed for tabular data to efficiently estimate the performance of candidate architectures, thereby obtaining a set of promising architectures. Subsequently, in the refinement phase, ATLAS leverages a fixed-budget search algorithm to schedule the training of the promising candidates, so as to accurately identify the optimal architecture. To jointly optimize the two phases for anytime NAS, we also devise a budget-aware coordinator that delivers high NAS performance within constraints. Experimental evaluations demonstrate that our ATLAS can obtain a good-performing architecture within any predefined time budget and return better architectures as and when a new time budget is made available. Overall, it reduces the search time on tabular data by up to 82.75x compared to existing NAS approaches.

Figures (18)

• Figure 1: The Overview of NAS.
• Figure 2: The empirical cumulative distribution (ECDF) of the training and validation AUC.
• Figure 3: Validation AUC vs. the number of trainable parameters.
• Figure 4: Benchmarking of the four search strategies using NAS-Bench-Tabular. The x-axis denotes the number of explored architectures, and the y-axis denotes the median value of the best AUC achieved across 100 runs.
• Figure 5: The relationship between the ExpressFlow score and the AUC of searched architecture.