100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances

TL;DR

This work tackles the challenge of predicting an unseen LLM’s performance on individual task instances without exhaustive evaluation. It introduces a generic assessor that combines instance-intrinsic features with LLM performance on a small reference set, trained across previously evaluated LLMs, to forecast per-instance outcomes for new LLMs. Empirical studies on HELM-Lite and the newly introduced KindsOfReasoning datasets show the generic assessor can rival LLM-specific assessors in-distribution and often outperform baselines, while highlighting reduced predictability out-of-distribution. The approach enables cheaper, instance-level reliability checks for LLM deployments and provides a public data resource for further benchmarking, though it acknowledges limits when distributions shift and calls for deeper analysis of what governs LLM predictability.

Abstract

Predicting the performance of LLMs on individual task instances is essential to ensure their reliability in high-stakes applications. To do so, a possibility is to evaluate the considered LLM on a set of task instances and train an assessor to predict its performance based on features of the instances. However, this approach requires evaluating each new LLM on a sufficiently large set of task instances to train an assessor specific to it. In this work, we leverage the evaluation results of previously tested LLMs to reduce the number of evaluations required to predict the performance of a new LLM. In practice, we propose to test the new LLM on a small set of reference instances and train a generic assessor which predicts the performance of the LLM on an instance based on the performance of the former on the reference set and features of the instance of interest. We conduct empirical studies on HELM-Lite and KindsOfReasoning, a collection of existing reasoning datasets that we introduce, where we evaluate all instruction-fine-tuned OpenAI models until the January 2024 version of GPT4. When predicting performance on instances with the same distribution as those used to train the generic assessor, we find this achieves performance comparable to the LLM-specific assessors trained on the full set of instances. Additionally, we find that randomly selecting the reference instances performs as well as some advanced selection methods we tested. For out of distribution, however, no clear winner emerges and the overall performance is worse, suggesting that the inherent predictability of LLMs is low.

Figures (8)

• Figure 1: Predictive performance (AUC) of the specific and generic assessor and a few baselines, for different splits of the KindsOfReasoning and HELM-Lite collections of datasets. Some combinations (for instance, the random selector on split 1 of KindsOfReasoning achieve AUC lower than the lower bound of the panels (0.4) and are hence hidden in the graph.
• Figure 2: AUC with different choices of instance-intrinsic features (OpenAI embeddings, Word2Vec, FastText and 1-gram), for different splits on KindsOfReasoning. For each split and feature, various classifiers were trained on $\mathcal{D}^\text{train}$ and the best according to its performance on $\mathcal{D}^\text{val}$ was selected; the panels report the performance of the latter on $\mathcal{D}^\text{val}$ and $\mathcal{D}^\text{test}$.
• Figure 3: AUC with different choices of instance-intrinsic features (OpenAI embeddings, Word2Vec, FastText and 1-gram), for different splits on HELM-Lite. For each split and feature, various classifiers were trained on $\mathcal{D}^\text{train}$ and the best according to its performance on $\mathcal{D}^\text{val}$ was selected; the panels report the performance of the latter on $\mathcal{D}^\text{val}$ and $\mathcal{D}^\text{test}$.
• Figure 4: AUC with increasing number of OpenAI embeddings for specific assessors trained on increasing number of OpenAI embeddings, for different splits on KindsOfReasoning. For each split and number of embeddings, various classifiers were trained on $\mathcal{D}^\text{train}$ and the best according to its performance on $\mathcal{D}^\text{val}$ was selected; the panels report the performance of the latter on $\mathcal{D}^\text{val}$ and $\mathcal{D}^\text{test}$.
• Figure 5: AUC with increasing number of OpenAI embeddings for specific assessors trained on increasing number of OpenAI embeddings, for different splits on HELM-Lite. For each split and number of embeddings, various classifiers were trained on $\mathcal{D}^\text{train}$ and the best according to its performance on $\mathcal{D}^\text{val}$ was selected; the panels report the performance of the latter on $\mathcal{D}^\text{val}$ and $\mathcal{D}^\text{test}$.