Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains

Zijie Wang, Farzana Rashid, Eduardo Blanco

TL;DR

This work tackles interpreting indirect yes-no answers in dialogues across multiple domains by first identifying yes-no questions with rule-based and BERT-based methods, then interpreting the answers using domain-adaptation techniques. It introduces three new benchmarks (Tennis, Movie, Air) with 300 indirect-answer instances each and a three-label scheme (Yes, No, Middle), learned via distant supervision and blended training. The study demonstrates that distant supervision combined with blending consistently improves cross-domain performance, achieving meaningful gains where synthetic data alone falls short, and shows that large-language models currently do not outperform a well-tuned RoBERTa-based approach for this task. The findings highlight practical pathways for adapting dialogue systems to interpret indirect answers while revealing limitations and directions for future work in data quality and multilingual extension.

Abstract

People often answer yes-no questions without explicitly saying yes, no, or similar polar keywords. Figuring out the meaning of indirect answers is challenging, even for large language models. In this paper, we investigate this problem working with dialogues from multiple domains. We present new benchmarks in three diverse domains: movie scripts, tennis interviews, and airline customer service. We present an approach grounded on distant supervision and blended training to quickly adapt to a new dialogue domain. Experimental results show that our approach is never detrimental and yields F1 improvements as high as 11-34%.

Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains

TL;DR

This work tackles interpreting indirect yes-no answers in dialogues across multiple domains by first identifying yes-no questions with rule-based and BERT-based methods, then interpreting the answers using domain-adaptation techniques. It introduces three new benchmarks (Tennis, Movie, Air) with 300 indirect-answer instances each and a three-label scheme (Yes, No, Middle), learned via distant supervision and blended training. The study demonstrates that distant supervision combined with blending consistently improves cross-domain performance, achieving meaningful gains where synthetic data alone falls short, and shows that large-language models currently do not outperform a well-tuned RoBERTa-based approach for this task. The findings highlight practical pathways for adapting dialogue systems to interpret indirect answers while revealing limitations and directions for future work in data quality and multilingual extension.

Abstract

People often answer yes-no questions without explicitly saying yes, no, or similar polar keywords. Figuring out the meaning of indirect answers is challenging, even for large language models. In this paper, we investigate this problem working with dialogues from multiple domains. We present new benchmarks in three diverse domains: movie scripts, tennis interviews, and airline customer service. We present an approach grounded on distant supervision and blended training to quickly adapt to a new dialogue domain. Experimental results show that our approach is never detrimental and yields F1 improvements as high as 11-34%.
Paper Structure (38 sections, 3 figures, 15 tables)

This paper contains 38 sections, 3 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Terminology and Existing Corpora
  3. Existing Corpora
  4. Related Works
  5. Identifying Yes-No Questions
  6. Collecting Yes-No Questions
  7. Corpora with Dialogue Acts Annotations
  8. Corpora without Dialogue Acts Annotations
  9. Classifiers to Identify Yes-No Questions
  10. Rule-based Classifier
  11. BERT Classifier
  12. Results and Analysis
  13. Interpreting Answers to Yes-No Questions
  14. Three New Benchmarks
  15. Inter-Annotator Agreements
  16. ...and 23 more sections

Figures (3)

  • Figure 1: Movie dialogue with three yes-no questions (A$_1$, A$_2$, and A$_3$). Answers are indirect as they do not include polar keywords (yes, no, etc.). In B$_1$ and B$_3$, the author declines to answer, whereas in B$_2$ the author indirectly answers no by minimizing the incident (injured requires loss of function, while hurt does not).
  • Figure 2: Heatmap of the inter-annotator agreements. The percentages are the average of three benchmarks (total: 900 instances). Most disagreements are between (1) Yes or No and (2) Middle.
  • Figure 3: Prompts used with GPT, Alpaca, and Llama.