Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches

Bhashithe Abeysinghe, Ruhan Circi

TL;DR

The paper addresses the difficulty of evaluating LLM-based chatbot applications by comparing automated metrics, traditional human evaluation, and LLM-based evaluators within a factored evaluation framework applied to an EdTalk chatbot. It demonstrates that factored, Bloom's taxonomy–driven analysis yields deeper insights into where LLMs falter, especially for Recall-type questions, and exposes reliability and agreement issues across evaluation methods. The study finds limited concordance between evaluation approaches, arguing that human evaluation remains essential in critical spaces, while LLM-based evaluators offer scalable but yet unreliable judgments. Overall, it advocates a robust, mixed evaluation pipeline that leverages factor-based human assessment and acknowledges the importance of data quality and diverse expertise to improve LLM applications in real-world settings.

Abstract

Chatbots have been an interesting application of natural language generation since its inception. With novel transformer based Generative AI methods, building chatbots have become trivial. Chatbots which are targeted at specific domains for example medicine and psychology are implemented rapidly. This however, should not distract from the need to evaluate the chatbot responses. Especially because the natural language generation community does not entirely agree upon how to effectively evaluate such applications. With this work we discuss the issue further with the increasingly popular LLM based evaluations and how they correlate with human evaluations. Additionally, we introduce a comprehensive factored evaluation mechanism that can be utilized in conjunction with both human and LLM-based evaluations. We present the results of an experimental evaluation conducted using this scheme in one of our chatbot implementations which consumed educational reports, and subsequently compare automated, traditional human evaluation, factored human evaluation, and factored LLM evaluation. Results show that factor based evaluation produces better insights on which aspects need to be improved in LLM applications and further strengthens the argument to use human evaluation in critical spaces where main functionality is not direct retrieval.

The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches

TL;DR

The paper addresses the difficulty of evaluating LLM-based chatbot applications by comparing automated metrics, traditional human evaluation, and LLM-based evaluators within a factored evaluation framework applied to an EdTalk chatbot. It demonstrates that factored, Bloom's taxonomy–driven analysis yields deeper insights into where LLMs falter, especially for Recall-type questions, and exposes reliability and agreement issues across evaluation methods. The study finds limited concordance between evaluation approaches, arguing that human evaluation remains essential in critical spaces, while LLM-based evaluators offer scalable but yet unreliable judgments. Overall, it advocates a robust, mixed evaluation pipeline that leverages factor-based human assessment and acknowledges the importance of data quality and diverse expertise to improve LLM applications in real-world settings.

Abstract

Chatbots have been an interesting application of natural language generation since its inception. With novel transformer based Generative AI methods, building chatbots have become trivial. Chatbots which are targeted at specific domains for example medicine and psychology are implemented rapidly. This however, should not distract from the need to evaluate the chatbot responses. Especially because the natural language generation community does not entirely agree upon how to effectively evaluate such applications. With this work we discuss the issue further with the increasingly popular LLM based evaluations and how they correlate with human evaluations. Additionally, we introduce a comprehensive factored evaluation mechanism that can be utilized in conjunction with both human and LLM-based evaluations. We present the results of an experimental evaluation conducted using this scheme in one of our chatbot implementations which consumed educational reports, and subsequently compare automated, traditional human evaluation, factored human evaluation, and factored LLM evaluation. Results show that factor based evaluation produces better insights on which aspects need to be improved in LLM applications and further strengthens the argument to use human evaluation in critical spaces where main functionality is not direct retrieval.
Paper Structure (11 sections, 2 figures, 7 tables)

This paper contains 11 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Evaluation procedures
  4. Automated evaluation
  5. Human evaluation
  6. LLM-based evaluation
  7. Results
  8. Discussion
  9. Prompts
  10. RAG Prompt
  11. LLM Evaluator Prompt

Figures (2)

  • Figure 1: Median of Likert scale ratings of each evaluator. Each spoke shows how an evaluator rated a response based on the question type from Blooms Taxonomy.
  • Figure 2: Screen capture of the EdTalk chatbot answering a question