A Two-Phase Recall-and-Select Framework for Fast Model Selection

TL;DR

Efficiently selecting a high-performing pre-trained network from large repositories is crucial for new tasks with limited data. The authors propose a two-phase recall-and-select framework that offline-builds a performance matrix and clusters models, then online executes a coarse recall via cluster representatives followed by a convergence-trend–based fine selection. By combining model clustering, proxy scores (via LEEP), and convergence-based filtering with successive halving, the approach achieves substantial speedups (roughly 2x–3x over successive halving and 5x–8x over brute-force) while maintaining near-peak accuracy across NLP and CV tasks. The method demonstrates strong scalability to more models and robust generalization to heterogeneous target tasks, offering a practical path for fast model selection in large model hubs.

Abstract

As the ubiquity of deep learning in various machine learning applications has amplified, a proliferation of neural network models has been trained and shared on public model repositories. In the context of a targeted machine learning assignment, utilizing an apt source model as a starting point typically outperforms the strategy of training from scratch, particularly with limited training data. Despite the investigation and development of numerous model selection strategies in prior work, the process remains time-consuming, especially given the ever-increasing scale of model repositories. In this paper, we propose a two-phase (coarse-recall and fine-selection) model selection framework, aiming to enhance the efficiency of selecting a robust model by leveraging the models' training performances on benchmark datasets. Specifically, the coarse-recall phase clusters models showcasing similar training performances on benchmark datasets in an offline manner. A light-weight proxy score is subsequently computed between this model cluster and the target dataset, which serves to recall a significantly smaller subset of potential candidate models in a swift manner. In the following fine-selection phase, the final model is chosen by fine-tuning the recalled models on the target dataset with successive halving. To accelerate the process, the final fine-tuning performance of each potential model is predicted by mining the model's convergence trend on the benchmark datasets, which aids in filtering lower performance models more earlier during fine-tuning. Through extensive experimentation on tasks covering natural language processing and computer vision, it has been demonstrated that the proposed methodology facilitates the selection of a high-performing model at a rate about 3x times faster than conventional baseline methods. Our code is available at https://github.com/plasware/two-phase-selection.

Figures (9)

• Figure 1: Fine-tuning performance of 44 and 25 pre-trained models on NLP task MNLI wang2018glue and CV task CC6204-Hackaton-Cu WahCUB_200_2011. The x and y-axis show pre-trained models' ID and their performances on the dataset, respectively. It's noted that for each dataset, the ids are sorted by the model accuracy desc.
• Figure 2: The framework of two-phase model selection: (a) performance matrix, (b) model clustering based on the performance matrix, and convergence trends mining by clustering convergence processes of a pre-trained model on benchmark datasets, (c) coarse-recall phase running recall strategy based on the proxy score computation between a model cluster and the target dataset, and (d) fine-selection phase fine-tunes the recalled models and filters poorly-performance models according to convergence trend. Both (a) and (b) are maintained offline and could be used for any new task.
• Figure 3: Top-10 models validation and test results on MNLI dataset. Model names ignore the repository name they belong to, the full names can be found in Table \ref{['models_nlpcv']} in Appendix. B two-phase-model-selection-tech-report.
• Figure 4: Validation&Test performance of the DoyyingFace/bert-asian-hate-tweets-asian-unclean-freeze-4 model on 30 datasets.
• Figure 5: The average accuracy comparison of recalled models