Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

F-Eval: Assessing Fundamental Abilities with Refined Evaluation Methods

Yu Sun, Keyu Chen, Shujie Wang, Peiji Li, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin

TL;DR

F-Eval introduces a bilingual benchmark to assess fundamental abilities of large language models—expression, commonsense, and logic—during pre-training, addressing gaps in instruction-following-centric evaluation. It combines 2211 English/Chinese instances across 15 sub-datasets with four evaluation methods, including new reference-free approaches and a self-adaptive normalization to integrate scores. Experimental results on 13 models show GPT-4.0 and GPT-3.5 leading, while open-source models lag behind; larger model sizes generally improve performance, particularly in logic, and F-Eval demonstrates stronger alignment with human judgments and greater discriminatory power than baselines. The work also analyzes normalization strategies and cross-dataset dynamics, providing a practical framework to monitor and advance fundamental abilities in LLMs.

Abstract

Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluation benchmarks are limited to assessing the instruction-following capabilities, overlooking the fundamental abilities that emerge during the pre-training stage. Previous subjective evaluation methods mainly reply on scoring by API models. However, in the absence of references, large models have shown limited ability to discern subtle differences. To bridge the gap, we propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. The tasks in F-Eval include multi-choice objective tasks, open-ended objective tasks, reference-based subjective tasks and reference-free subjective tasks. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models. We conduct evaluations on 13 advanced LLMs. Results show that our evaluation methods show higher correlation coefficients and larger distinction than other evaluators. Additionally, we discuss the influence of different model sizes, dimensions, and normalization methods. We anticipate that F-Eval will facilitate the study of LLMs' fundamental abilities.

F-Eval: Assessing Fundamental Abilities with Refined Evaluation Methods

TL;DR

F-Eval introduces a bilingual benchmark to assess fundamental abilities of large language models—expression, commonsense, and logic—during pre-training, addressing gaps in instruction-following-centric evaluation. It combines 2211 English/Chinese instances across 15 sub-datasets with four evaluation methods, including new reference-free approaches and a self-adaptive normalization to integrate scores. Experimental results on 13 models show GPT-4.0 and GPT-3.5 leading, while open-source models lag behind; larger model sizes generally improve performance, particularly in logic, and F-Eval demonstrates stronger alignment with human judgments and greater discriminatory power than baselines. The work also analyzes normalization strategies and cross-dataset dynamics, providing a practical framework to monitor and advance fundamental abilities in LLMs.

Abstract

Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluation benchmarks are limited to assessing the instruction-following capabilities, overlooking the fundamental abilities that emerge during the pre-training stage. Previous subjective evaluation methods mainly reply on scoring by API models. However, in the absence of references, large models have shown limited ability to discern subtle differences. To bridge the gap, we propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. The tasks in F-Eval include multi-choice objective tasks, open-ended objective tasks, reference-based subjective tasks and reference-free subjective tasks. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models. We conduct evaluations on 13 advanced LLMs. Results show that our evaluation methods show higher correlation coefficients and larger distinction than other evaluators. Additionally, we discuss the influence of different model sizes, dimensions, and normalization methods. We anticipate that F-Eval will facilitate the study of LLMs' fundamental abilities.
Paper Structure (104 sections, 6 equations, 7 figures, 11 tables)

This paper contains 104 sections, 6 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark
  4. Data Collection
  5. Expression
  6. Commonsense
  7. Logic
  8. Evaluation Methods
  9. Rule-based Evaluation
  10. Probability Evaluation
  11. Assistant-Tool Evaluation
  12. API Evaluation
  13. Results Normalization
  14. Experiments
  15. Settings
  16. ...and 89 more sections

Figures (7)

  • Figure 1: Prompts with instructions limit the capabilities of the base model (middle), which can be resolved either by removing instructions (upper) or by further SFT on the base model (lower).
  • Figure 2: The examples of each data format.
  • Figure 3: Overall composition of F-Eval.
  • Figure 4: Main results of F-Eval across 13 LLMs.
  • Figure 5: The distribution of the scores computed by GPT4.0 and F-Eval.
  • ...and 2 more figures