Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability

Xinyu Hu, Li Lin, Mingqi Gao, Xunjian Yin, Xiaojun Wan

TL;DR

Themis presents a reference-free NLG evaluation language model built on a purpose-built NLG-Eval corpus that aggregates 58 datasets across 9 tasks (~0.5M samples) annotated by humans and GPT-4. It introduces multi-perspective consistency verification and rating-guided preference alignment to train a reliable 8B LLM tailored for NLG evaluation, enabling flexible, interpretable analyses across diverse tasks and unseen domains. Empirical results show Themis achieves superior correlation with human judgments across six standard NLG benchmarks, surpassing many baselines and matching or approaching GPT-4 while remaining offline and reproducible. The work provides a valuable data/resource release and a scalable framework for high-quality, reference-free evaluation in real-world NLG systems.

Abstract

The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and model-based metrics. However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility. Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus NLG-Eval with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field. Furthermore, we propose Themis, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods. Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.

Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability

TL;DR

Themis presents a reference-free NLG evaluation language model built on a purpose-built NLG-Eval corpus that aggregates 58 datasets across 9 tasks (~0.5M samples) annotated by humans and GPT-4. It introduces multi-perspective consistency verification and rating-guided preference alignment to train a reliable 8B LLM tailored for NLG evaluation, enabling flexible, interpretable analyses across diverse tasks and unseen domains. Empirical results show Themis achieves superior correlation with human judgments across six standard NLG benchmarks, surpassing many baselines and matching or approaching GPT-4 while remaining offline and reproducible. The work provides a valuable data/resource release and a scalable framework for high-quality, reference-free evaluation in real-world NLG systems.

Abstract

The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and model-based metrics. However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility. Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus NLG-Eval with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field. Furthermore, we propose Themis, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods. Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.
Paper Structure (31 sections, 6 equations, 3 figures, 26 tables)

This paper contains 31 sections, 6 equations, 3 figures, 26 tables.

Table of Contents

  1. Introduction
  2. NLG-Eval Corpus
  3. Definitions
  4. Data Collection
  5. GPT-4 Annotations
  6. Methodology
  7. Diversity and Balance
  8. Multi-perspective Consistency Verification
  9. Self-Consistency
  10. Cross-Validation
  11. Evaluation Inspection
  12. Preference Alignment
  13. Preference Construction
  14. Rating-guided DPO
  15. Experiments
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Themis is capable of evaluating various NLG tasks based on flexible evaluation aspects and criteria without references and providing corresponding analyses along with the evaluation ratings.
  • Figure 2: The NLG evaluation tasks and corresponding evaluation aspects in our NLG-Eval corpus.
  • Figure 3: A simplified example from the hsplit dataset for the text simplification task.