Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents

Shiwei Wu, Chen Zhang, Yan Gao, Qimeng Wang, Tong Xu, Yao Hu, Enhong Chen

TL;DR

The paper presents InsCoQA, a benchmark for conversational QA over multi-document instructional content, addressing the gap left by single-document datasets. It collects 13.9k instructional conversations from Xiaohongshu across 13 domains and converts them into multi-turn dialogs that require retrieving, interpreting, and summarizing procedural guidance from multiple sources. To evaluate models, the authors introduce InsEval, an LLM-assisted evaluator that combines Judge Score, Task Completion Rate, and ROUGE-L text-matching metrics. Experimental results show GPT-4 generally leads among tested LLMs, with smaller evaluators able to approximate relative performance, highlighting InsCoQA and InsEval as valuable tools for assessing real-world, instruction-focused conversational capabilities of LLMs.

Abstract

Instructional documents are rich sources of knowledge for completing various tasks, yet their unique challenges in conversational question answering (CQA) have not been thoroughly explored. Existing benchmarks have primarily focused on basic factual question-answering from single narrative documents, making them inadequate for assessing a model`s ability to comprehend complex real-world instructional documents and provide accurate step-by-step guidance in daily life. To bridge this gap, we present InsCoQA, a novel benchmark tailored for evaluating large language models (LLMs) in the context of CQA with instructional documents. Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents, reflecting the intricate and multi-faceted nature of real-world instructional tasks. Additionally, to comprehensively assess state-of-the-art LLMs on the InsCoQA benchmark, we propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.

Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents

TL;DR

The paper presents InsCoQA, a benchmark for conversational QA over multi-document instructional content, addressing the gap left by single-document datasets. It collects 13.9k instructional conversations from Xiaohongshu across 13 domains and converts them into multi-turn dialogs that require retrieving, interpreting, and summarizing procedural guidance from multiple sources. To evaluate models, the authors introduce InsEval, an LLM-assisted evaluator that combines Judge Score, Task Completion Rate, and ROUGE-L text-matching metrics. Experimental results show GPT-4 generally leads among tested LLMs, with smaller evaluators able to approximate relative performance, highlighting InsCoQA and InsEval as valuable tools for assessing real-world, instruction-focused conversational capabilities of LLMs.

Abstract

Instructional documents are rich sources of knowledge for completing various tasks, yet their unique challenges in conversational question answering (CQA) have not been thoroughly explored. Existing benchmarks have primarily focused on basic factual question-answering from single narrative documents, making them inadequate for assessing a model`s ability to comprehend complex real-world instructional documents and provide accurate step-by-step guidance in daily life. To bridge this gap, we present InsCoQA, a novel benchmark tailored for evaluating large language models (LLMs) in the context of CQA with instructional documents. Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents, reflecting the intricate and multi-faceted nature of real-world instructional tasks. Additionally, to comprehensively assess state-of-the-art LLMs on the InsCoQA benchmark, we propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
Paper Structure (19 sections, 4 figures, 4 tables)

This paper contains 19 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Constructing Conversational QA Dataset for Multiple Instructional Documents
  4. Collecting Instructional Documents for Popular User Queries
  5. Formatting to Conversational QA Dialog
  6. Dataset analysis
  7. Data Composition
  8. Data Statistics
  9. Data Characteristics
  10. InsEval: Comprehensive Evaluation for InsCoQA benchmark
  11. LLM-assisted Evaluation Metrics
  12. Text-matching Evaluation Metrics
  13. Experimental Results on LLMs
  14. Conclusion
  15. Query Filtering Prompt
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Comparison of Conversational Question Answering (CQA) Samples. Our InsCoQA dataset provides more complex procedural guidance derived from multiple instructional documents, addressing intricate, real-world tasks. In contrast, the previous CQA sample involves basic narrative elements and factual recall, focusing on simple, single-document information.
  • Figure 2: An instructional example from the InsCoQA dataset, where the conversational QA is closely aligned with the referenced instructional documents. This sample highlights key features such as multi-document references and a focus on providing procedural instructions for real-world tasks.
  • Figure 3: Pipeline for constructing the InsCoQA dataset. We begin by retrieving multiple instructional documents relevant to the instructional-intended user query, then rewriting them into multi-turn instructional conversational Q&A. The responses are further summarized into procedural instructions. The InsCoQA dataset is finally constructed after a thorough human check.
  • Figure 4: Domain distribution of InsCoQA dataset.