HyperFast: Instant Classification for Tabular Data

TL;DR

HyperFast proposes a meta trained hypernetwork that, from a dataset support set, instantly outputs the weights of a lightweight main network for tabular classification in a single forward pass. The method pairs initial data transformations with permutation invariant hypernetwork modules to generate layer weights, enabling fast inference without conventional training and allowing ensembling and fine tuning for accuracy gains. Across OpenML and genomics data, HyperFast delivers competitive performance while dramatically reducing compute time, often outperforming AutoML systems at similar budgets and surpassing tabular deep learning models in many settings. This approach promises substantial reductions in the computational burden of deep learning for tabular data and opens avenues for edge deployment and rapid model iteration, with open source code and a scikit learn like interface.

Abstract

Training deep learning models and performing hyperparameter tuning can be computationally demanding and time-consuming. Meanwhile, traditional machine learning methods like gradient-boosting algorithms remain the preferred choice for most tabular data applications, while neural network alternatives require extensive hyperparameter tuning or work only in toy datasets under limited settings. In this paper, we introduce HyperFast, a meta-trained hypernetwork designed for instant classification of tabular data in a single forward pass. HyperFast generates a task-specific neural network tailored to an unseen dataset that can be directly used for classification inference, removing the need for training a model. We report extensive experiments with OpenML and genomic data, comparing HyperFast to competing tabular data neural networks, traditional ML methods, AutoML systems, and boosting machines. HyperFast shows highly competitive results, while being significantly faster. Additionally, our approach demonstrates robust adaptability across a variety of classification tasks with little to no fine-tuning, positioning HyperFast as a strong solution for numerous applications and rapid model deployment. HyperFast introduces a promising paradigm for fast classification, with the potential to substantially decrease the computational burden of deep learning. Our code, which offers a scikit-learn-like interface, along with the trained HyperFast model, can be found at https://github.com/AI-sandbox/HyperFast.

Figures (9)

• Figure 1: (left) HyperFast framework. (right) Architecture detail. Each hypernetwork module receives representations of the support set of batch size ($bs$) samples. The modules $l \in [1, L-1]$ compress the representations into a single embedding of hypernetwork hidden size ($h_{\text{hn}}$) to then generate the main network weights $\theta_{\text{main}_l}$. Module $L$ summarizes the representations per class with embeddings of main hidden size$(h_{\text{m}})+1$, directly obtaining the weights of the last classification layer $\theta_{\text{main}_L}$.
• Figure 2: Runtime (fit + predict) vs. performance and average rank for given runtime budgets on the mini-test (small-sized version of the 15 meta-test datasets with $\leq$ 1000 training examples, $\leq$ 100 features and $\leq$ 10 classes restrictions).
• Figure 3: Runtime (fit + predict) vs. performance and average rank for given runtime budgets on the big test: 15 large/medium-sized meta-datasets.
• Figure 4: Classifiers comparison with the decision boundaries for toy binary classification datasets.
• Figure 5: Performance as a function of runtime for different approaches to fully leverage all training data in the generation of the final inference model with HyperFast. Batch sizes considered in a single forward pass: [64, 128, 256, 512, 784, 1024, 2048, 4096, 8192, 16384]. Number of members considered in options involving ensembling or stacking: [1, 2, 3, 4, 8, 10, 15, 20, 32]. Optimization steps trials: [0, 2, 3, 4, 8, 16, 32, 64, 256, 1024, 4096].