Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop

Anum Afzal, Alexander Kowsik, Rajna Fani, Florian Matthes

TL;DR

The paper addresses building and evaluating a retrieval-augmented QA chatbot for HR using industrial data from SAP SE, with domain experts in the loop across data collection, prompting, and assessment. It compares two retrieval streams (DPR and OpenAI vector search) and two generation strategies (LongT5 fine-tuning and OpenAI prompts), finding GPT-4 offers superior generation while DPR yields stronger retrieval performance on HR data. A key finding is that traditional reference-based metrics correlate poorly with human judgments for generative LLM outputs, whereas reference-free metrics like G-Eval and Prometheus align more closely on average, though model-dependent. The work demonstrates practical feasibility and limitations of deploying RAG-powered HR support tools in real-world settings and highlights the importance of human expertise for reliable evaluation and continuous improvement.

Abstract

Large Language Models have found application in various mundane and repetitive tasks including Human Resource (HR) support. We worked with the domain experts of SAP SE to develop an HR support chatbot as an efficient and effective tool for addressing employee inquiries. We inserted a human-in-the-loop in various parts of the development cycles such as dataset collection, prompt optimization, and evaluation of generated output. By enhancing the LLM-driven chatbot's response quality and exploring alternative retrieval methods, we have created an efficient, scalable, and flexible tool for HR professionals to address employee inquiries effectively. Our experiments and evaluation conclude that GPT-4 outperforms other models and can overcome inconsistencies in data through internal reasoning capabilities. Additionally, through expert analysis, we infer that reference-free evaluation metrics such as G-Eval and Prometheus demonstrate reliability closely aligned with that of human evaluation.

Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop

TL;DR

The paper addresses building and evaluating a retrieval-augmented QA chatbot for HR using industrial data from SAP SE, with domain experts in the loop across data collection, prompting, and assessment. It compares two retrieval streams (DPR and OpenAI vector search) and two generation strategies (LongT5 fine-tuning and OpenAI prompts), finding GPT-4 offers superior generation while DPR yields stronger retrieval performance on HR data. A key finding is that traditional reference-based metrics correlate poorly with human judgments for generative LLM outputs, whereas reference-free metrics like G-Eval and Prometheus align more closely on average, though model-dependent. The work demonstrates practical feasibility and limitations of deploying RAG-powered HR support tools in real-world settings and highlights the importance of human expertise for reliable evaluation and continuous improvement.

Abstract

Large Language Models have found application in various mundane and repetitive tasks including Human Resource (HR) support. We worked with the domain experts of SAP SE to develop an HR support chatbot as an efficient and effective tool for addressing employee inquiries. We inserted a human-in-the-loop in various parts of the development cycles such as dataset collection, prompt optimization, and evaluation of generated output. By enhancing the LLM-driven chatbot's response quality and exploring alternative retrieval methods, we have created an efficient, scalable, and flexible tool for HR professionals to address employee inquiries effectively. Our experiments and evaluation conclude that GPT-4 outperforms other models and can overcome inconsistencies in data through internal reasoning capabilities. Additionally, through expert analysis, we infer that reference-free evaluation metrics such as G-Eval and Prometheus demonstrate reliability closely aligned with that of human evaluation.
Paper Structure (39 sections, 2 figures, 9 tables)

This paper contains 39 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Corpus
  3. Dataset Collection
  4. Dataset Statistics
  5. Methodology
  6. Retriever
  7. Dense Passage Retriever (BERT)
  8. Vector Search (OpenAI)
  9. NLG Module
  10. LongT5 (Fine-tuning driven)
  11. OpenAI Models (Prompt driven)
  12. Evaluation Framework
  13. Retriever Evaluation
  14. Human Evaluation Setup
  15. Reference-based Metrics
  16. ...and 24 more sections

Figures (2)

  • Figure 1: Distribution over the number of tokens of all unique articles in our HR dataset.
  • Figure 2: Block diagram of the methodology introduced in our paper, illustrating baseline and Open AI models, highlighting the role of the human-in-the-loop during development