Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification

Priya Mishra, Suraj Racha, Kaustubh Ponkshe, Adit Akarsh, Ganesh Ramakrishnan

TL;DR

This work presents a novel framework for asking guided questions to further progress a partial information, and demonstrates that GUIDEQ outperforms other LLM-based baselines, yielding improved F1-Score through the accurate collection of relevant further information.

Abstract

Question Answering (QA) is an important part of tasks like text classification through information gathering. These are finding increasing use in sectors like healthcare, customer support, legal services, etc., to collect and classify responses into actionable categories. LLMs, although can support QA systems, they face a significant challenge of insufficient or missing information for classification. Although LLMs excel in reasoning, the models rely on their parametric knowledge to answer. However, questioning the user requires domain-specific information aiding to collect accurate information. Our work, GUIDEQ, presents a novel framework for asking guided questions to further progress a partial information. We leverage the explainability derived from the classifier model for along with LLMs for asking guided questions to further enhance the information. This further information helps in more accurate classification of a text. GUIDEQ derives the most significant key-words representative of a label using occlusions. We develop GUIDEQ's prompting strategy for guided questions based on the top-3 classifier label outputs and the significant words, to seek specific and relevant information, and classify in a targeted manner. Through our experimental results, we demonstrate that GUIDEQ outperforms other LLM-based baselines, yielding improved F1-Score through the accurate collection of relevant further information. We perform various analytical studies and also report better question quality compared to our method.

GUIDEQ: Framework for Guided Questioning for progressive informational collection and classification

TL;DR

This work presents a novel framework for asking guided questions to further progress a partial information, and demonstrates that GUIDEQ outperforms other LLM-based baselines, yielding improved F1-Score through the accurate collection of relevant further information.

Abstract

Question Answering (QA) is an important part of tasks like text classification through information gathering. These are finding increasing use in sectors like healthcare, customer support, legal services, etc., to collect and classify responses into actionable categories. LLMs, although can support QA systems, they face a significant challenge of insufficient or missing information for classification. Although LLMs excel in reasoning, the models rely on their parametric knowledge to answer. However, questioning the user requires domain-specific information aiding to collect accurate information. Our work, GUIDEQ, presents a novel framework for asking guided questions to further progress a partial information. We leverage the explainability derived from the classifier model for along with LLMs for asking guided questions to further enhance the information. This further information helps in more accurate classification of a text. GUIDEQ derives the most significant key-words representative of a label using occlusions. We develop GUIDEQ's prompting strategy for guided questions based on the top-3 classifier label outputs and the significant words, to seek specific and relevant information, and classify in a targeted manner. Through our experimental results, we demonstrate that GUIDEQ outperforms other LLM-based baselines, yielding improved F1-Score through the accurate collection of relevant further information. We perform various analytical studies and also report better question quality compared to our method.

Paper Structure

This paper contains 29 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Posthoc Explanations and LLMs
  4. Task oriented dialogues
  5. Information-seeking questions using LLMs
  6. In-context learning
  7. GuideQ - Methodology
  8. Overview
  9. Classifier Finetuning
  10. Keywords learning
  11. Explainability-Driven Question Generation
  12. Experimental Setup
  13. Datasets
  14. Baselines
  15. Evaluation
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Illustration of partial information by user followed by a specific guided question
  • Figure 2: (A) Overall working framework of GuideQ to leverage LLM and label explainability for asking guided question. (B) Details of the prompting strategy used. (C) Final classification along with incremental information.
  • Figure 3: Multiturn Results: F1-Scores for three turn question answering on CNEWS datatset.