Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusion-Eval: Integrating Assistant Evaluators with LLMs

Lei Shu, Nevan Wichers, Liangchen Luo, Yun Zhu, Yinxiao Liu, Jindong Chen, Lei Meng

TL;DR

Fusion-Eval introduces an LLM-based framework to fuse multiple assistant evaluators for evaluating NLG in a reference-free setting, achieving higher correlations with human judgments than baselines on SummEval and TopicalChat. It offers two prompting schemes, FE-NoPlan and FE with Plan, and a prompt-execution step that averages eight predictions to derive final scores. The method leverages several assistant evaluators (e.g., NLI, BLEURT, SumBLEURT, PaLM2 Prob) and demonstrates that planning prompts can improve performance, while the aggregated approach consistently outperforms simple score aggregation. The results suggest that integrating diverse evaluators with an LLM can enhance evaluation reliability and scalability for well-established text generation tasks.

Abstract

Evaluating natural language systems poses significant challenges, particularly in the realms of natural language understanding and high-level reasoning. In this paper, we introduce 'Fusion-Eval', an innovative approach that leverages Large Language Models (LLMs) to integrate insights from various assistant evaluators. The LLM is given the example to evaluate along with scores from the assistant evaluators. Each of these evaluators specializes in assessing distinct aspects of responses. Fusion-Eval achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. These results highlight Fusion-Eval's significant potential in the realm of natural language system evaluation.

Fusion-Eval: Integrating Assistant Evaluators with LLMs

TL;DR

Fusion-Eval introduces an LLM-based framework to fuse multiple assistant evaluators for evaluating NLG in a reference-free setting, achieving higher correlations with human judgments than baselines on SummEval and TopicalChat. It offers two prompting schemes, FE-NoPlan and FE with Plan, and a prompt-execution step that averages eight predictions to derive final scores. The method leverages several assistant evaluators (e.g., NLI, BLEURT, SumBLEURT, PaLM2 Prob) and demonstrates that planning prompts can improve performance, while the aggregated approach consistently outperforms simple score aggregation. The results suggest that integrating diverse evaluators with an LLM can enhance evaluation reliability and scalability for well-established text generation tasks.

Abstract

Evaluating natural language systems poses significant challenges, particularly in the realms of natural language understanding and high-level reasoning. In this paper, we introduce 'Fusion-Eval', an innovative approach that leverages Large Language Models (LLMs) to integrate insights from various assistant evaluators. The LLM is given the example to evaluate along with scores from the assistant evaluators. Each of these evaluators specializes in assessing distinct aspects of responses. Fusion-Eval achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. These results highlight Fusion-Eval's significant potential in the realm of natural language system evaluation.
Paper Structure (17 sections, 1 figure, 4 tables)

This paper contains 17 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. (1) Fusion-Eval without Plan (FE-NoPlan)
  4. (2) Fusion-Eval with Plan (FE)
  5. Prompt Execution
  6. Experiment
  7. Experiment Setting
  8. Baselines
  9. Result Analysis
  10. Fusion-Eval Performance
  11. Fusion-Eval Execution Time
  12. Correlations between Fusion-Eval And Assistant Evaluators
  13. Conclusion
  14. Limitation and Future Work
  15. Appendix
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Workflow of Fusion-Eval with Plan: Starting from the left, a query initiates the generation of a plan by the LLM. Once the plan is obtained, it is concatenated with the template. The template placeholders are filled in for each test example along with its specific assistant evaluators' scores. This complete prompt is then used to obtain the Fusion-Eval evaluation score from the LLM.