Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

"There Is No Such Thing as a Dumb Question," But There Are Good Ones

Minjung Shin, Donghyun Kim, Jeh-Kwang Ryu

TL;DR

This paper addresses the lack of systematic evaluation for question quality in human-AI interaction by proposing a two-dimensional framework of appropriateness and effectiveness, operationalized via a rubric-based, semi-adaptive scoring system. The metric decomposes into sub-components (Cohesion, Answerability, Respectfulness) for appropriateness and (Clarity, Coherence, Informativeness) for effectiveness, with dynamic context variables (answerer and goal) guiding assessments. Validation and empirical testing on CAUS and SQUARE demonstrate the framework's ability to differentiate high-quality questions from problematical ones across real and adversarial contexts, and illustrate how context-sensitive scoring yields interpretable, actionable insights for Question Generation and evaluation. The approach offers a practical, explainable method to improve human-AI questioning and informs future research in educational and safety-critical domains.

Abstract

Questioning has become increasingly crucial for both humans and artificial intelligence, yet there remains limited research comprehensively assessing question quality. In response, this study defines good questions and presents a systematic evaluation framework. We propose two key evaluation dimensions: appropriateness (sociolinguistic competence in context) and effectiveness (strategic competence in goal achievement). Based on these foundational dimensions, a rubric-based scoring system was developed. By incorporating dynamic contextual variables, our evaluation framework achieves structure and flexibility through semi-adaptive criteria. The methodology was validated using the CAUS and SQUARE datasets, demonstrating the ability of the framework to access both well-formed and problematic questions while adapting to varied contexts. As we establish a flexible and comprehensive framework for question evaluation, this study takes a significant step toward integrating questioning behavior with structured analytical methods grounded in the intrinsic nature of questioning.

"There Is No Such Thing as a Dumb Question," But There Are Good Ones

TL;DR

This paper addresses the lack of systematic evaluation for question quality in human-AI interaction by proposing a two-dimensional framework of appropriateness and effectiveness, operationalized via a rubric-based, semi-adaptive scoring system. The metric decomposes into sub-components (Cohesion, Answerability, Respectfulness) for appropriateness and (Clarity, Coherence, Informativeness) for effectiveness, with dynamic context variables (answerer and goal) guiding assessments. Validation and empirical testing on CAUS and SQUARE demonstrate the framework's ability to differentiate high-quality questions from problematical ones across real and adversarial contexts, and illustrate how context-sensitive scoring yields interpretable, actionable insights for Question Generation and evaluation. The approach offers a practical, explainable method to improve human-AI questioning and informs future research in educational and safety-critical domains.

Abstract

Questioning has become increasingly crucial for both humans and artificial intelligence, yet there remains limited research comprehensively assessing question quality. In response, this study defines good questions and presents a systematic evaluation framework. We propose two key evaluation dimensions: appropriateness (sociolinguistic competence in context) and effectiveness (strategic competence in goal achievement). Based on these foundational dimensions, a rubric-based scoring system was developed. By incorporating dynamic contextual variables, our evaluation framework achieves structure and flexibility through semi-adaptive criteria. The methodology was validated using the CAUS and SQUARE datasets, demonstrating the ability of the framework to access both well-formed and problematic questions while adapting to varied contexts. As we establish a flexible and comprehensive framework for question evaluation, this study takes a significant step toward integrating questioning behavior with structured analytical methods grounded in the intrinsic nature of questioning.
Paper Structure (20 sections, 4 figures, 4 tables)

This paper contains 20 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Question Evaluation in NLP
  3. Development and Validation of Question Evaluation Metric
  4. Metric Development
  5. Sub-Components of Appropriateness
  6. Sub-Components of Effectiveness
  7. Rubric-Based Scoring System
  8. Setting Scores and Descriptions
  9. Configuring Context Variables
  10. Validity Test of the Evaluation Metric
  11. Methods
  12. Dataset Selection and Processing
  13. CAUS Dataset
  14. SQAURE Dataset
  15. Evaluation Method
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Original script for validity test (FQ: Follow-up Question; FA: Follow-up Answer)
  • Figure 2: Result of the validity test. Colored lines represent three distinct questions. Two charts demonstrate information acquisition (Left) and social interaction (Right) contexts.
  • Figure 3: Evaluation of the CAUS dataset. The three radial graphs display the analysis results of 50 questions each: (Left) the first generated, (Middle) the third generated, and (Right) the fifth generated questions for given scenes.
  • Figure 4: Evaluation of the SQUARE dataset. The three radial graphs show the results of the analysis for 50 questions each: (Left) contentious questions, (Middle) questions asking for ethical judgments, (Right) predictive questions.