RevisEval: Improving LLM-as-a-Judge via Response-Adapted References

TL;DR

RevisEval is a novel text generation evaluation paradigm via the response-adapted references that outperforms traditional reference-free and reference-based evaluation paradigms that use LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.

Abstract

With significant efforts in recent studies, LLM-as-a-Judge has become a cost-effective alternative to human evaluation for assessing text generation quality in a wide range of tasks. However, there still remains a reliability gap between LLM-as-a-Judge and human evaluation. One important reason is the lack of guided oracles in the evaluation process. Motivated by the role of reference pervasively used in classic text evaluation, we introduce RevisEval, a novel text generation evaluation paradigm via the response-adapted references. RevisEval is driven by the key observation that an ideal reference should maintain the necessary relevance to the response to be evaluated. Specifically, RevisEval leverages the text revision capabilities of large language models (LLMs) to adaptively revise the response, then treat the revised text as the reference (response-adapted reference) for the subsequent evaluation. Extensive experiments demonstrate that RevisEval outperforms traditional reference-free and reference-based evaluation paradigms that use LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks. More importantly, our response-adapted references can further boost the classical text metrics, e.g., BLEU and BERTScore, compared to traditional references and even rival the LLM-as-a-Judge. A detailed analysis is also conducted to confirm RevisEval's effectiveness in bias reduction, the impact of inference cost, and reference relevance.

Figures (11)

• Figure 1: Performance comparison of reference-free and reference-based evaluation paradigms across different similarity groups in MT-Bench, using GPT-4-as-a-Judge. In the reference-based evaluation, the GPT-4 direct response is used as the reference, and the evaluated response with a higher BERTScore with the reference is regarded as the preferred one. As the similarity between the reference and the response increases, the human agreement accuracy of the reference-based evaluation significantly improves, while the reference-free evaluation maintains relatively consistent performance across all similarity levels.
• Figure 2: Illustration of our proposed RevisEval. Given an instance ($x,y,a$), we use RevisEval to assess $y$ in rubric $a$. In RevisEval, (i) reviser generates a response-adapted reference $r^\star$ by revising the $y$ to enhance the (ii) following LLM-as-a-Judge, even classic text metrics. Here, ... represents retained segments during the generating response-adapted reference process.
• Figure 3: Comparative analysis of reference-based metrics performance using references generated by Human/GPT-4 and RevisEval on NLG and instruction following benchmarks. RevisEval greatly enhance traditional ref-based metrics, even achieving them comparable to GPT-4-as-a-Judge.
• Figure 4: The prompt of reference-free pairwise comparison evaluation.
• Figure 5: The prompt of reference-based pairwise comparison evaluation.