What is the best model? Application-driven Evaluation for Large Language Models

TL;DR

The paper addresses the challenge of selecting the most cost-effective LLM for practical tasks by introducing A-Eval, an application-driven evaluation benchmark. It constructs a 678-question non-choice QA dataset across five task categories and three difficulty levels, and employs both automatic scoring (with a SOTA scorer) and expert evaluation to assess eight Qwen1.5-Chat scales (0.5B–110B). The authors uncover size- and difficulty-related laws in performance and propose a three-step, intersection-based approach to choose the smallest model meeting a desired accuracy, providing actionable guidance for real-world deployment. This work offers a concrete, application-focused framework to guide model selection and sets the stage for automated, scalable evaluation and expansion of the benchmark.

Abstract

General large language models enhanced with supervised fine-tuning and reinforcement learning from human feedback are increasingly popular in academia and industry as they generalize foundation models to various practical tasks in a prompt manner. To assist users in selecting the best model in practical application scenarios, i.e., choosing the model that meets the application requirements while minimizing cost, we introduce A-Eval, an application-driven LLMs evaluation benchmark for general large language models. First, we categorize evaluation tasks into five main categories and 27 sub-categories from a practical application perspective. Next, we construct a dataset comprising 678 question-and-answer pairs through a process of collecting, annotating, and reviewing. Then, we design an objective and effective evaluation method and evaluate a series of LLMs of different scales on A-Eval. Finally, we reveal interesting laws regarding model scale and task difficulty level and propose a feasible method for selecting the best model. Through A-Eval, we provide clear empirical and engineer guidance for selecting the best model, reducing barriers to selecting and using LLMs and promoting their application and development. Our benchmark is publicly available at https://github.com/UnicomAI/DataSet/tree/main/TestData/GeneralAbility.

Figures (8)

• Figure 1: A greater number of cortical neurons leads to a higher intelligence level.
• Figure 2: Application-driven evaluation tasks taxonomy.
• Figure 3: Data distribution of A-Eval dataset.
• Figure 4: An automatic evaluation example.
• Figure 5: Accuracy of models with different scales. (a) The average accuracy of models of varying scales across all tasks and difficulty levels. The dotted line represents expert evaluation results, while the solid lines represent automatic evaluation results with different scoring thresholds $T$. (b) The average accuracy of models of varying scales on easy, medium, and hard data. The dotted line represents expert evaluation results, and the solid lines depict automatic evaluation results using scoring thresholds of 90 and 60, respectively.