Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hey, wait a minute: on at-issue sensitivity in Language Models

Sanghee J. Kim, Kanishka Misra

TL;DR

The paper tackles the challenge of evaluating dialogue naturalness in language models by grounding the assessment in the linguistic notion of at-issueness and introducing DGRC, a minimal-pair–inspired method that samples model continuations to test discourse sensitivity via log-probabilities. DGRC divides an utterance, generates continuations for each part, recombines them, and compares likelihoods to reveal whether models preferentially target at-issue content. Across two experiments, the authors find that LMs prefer at-issue continuations, a tendency amplified in instruct-tuned models, and that digression cues can modulate this preference, especially within ARC structures. The approach provides a bias-resilient, scalable framework for analyzing dialogue dynamics and offers insights into how instruct-tuning and discourse cues shape naturalistic model behavior, with implications for improving conversational quality.

Abstract

Evaluating the naturalness of dialogue in language models (LMs) is not trivial: notions of 'naturalness' vary, and scalable quantitative metrics remain limited. This study leverages the linguistic notion of 'at-issueness' to assess dialogue naturalness and introduces a new method: Divide, Generate, Recombine, and Compare (DGRC). DGRC (i) divides a dialogue as a prompt, (ii) generates continuations for subparts using LMs, (iii) recombines the dialogue and continuations, and (iv) compares the likelihoods of the recombined sequences. This approach mitigates bias in linguistic analyses of LMs and enables systematic testing of discourse-sensitive behavior. Applying DGRC, we find that LMs prefer to continue dialogue on at-issue content, with this effect enhanced in instruct-tuned models. They also reduce their at-issue preference when relevant cues (e.g., "Hey, wait a minute") are present. Although instruct-tuning does not further amplify this modulation, the pattern reflects a hallmark of successful dialogue dynamics.

Hey, wait a minute: on at-issue sensitivity in Language Models

TL;DR

The paper tackles the challenge of evaluating dialogue naturalness in language models by grounding the assessment in the linguistic notion of at-issueness and introducing DGRC, a minimal-pair–inspired method that samples model continuations to test discourse sensitivity via log-probabilities. DGRC divides an utterance, generates continuations for each part, recombines them, and compares likelihoods to reveal whether models preferentially target at-issue content. Across two experiments, the authors find that LMs prefer at-issue continuations, a tendency amplified in instruct-tuned models, and that digression cues can modulate this preference, especially within ARC structures. The approach provides a bias-resilient, scalable framework for analyzing dialogue dynamics and offers insights into how instruct-tuning and discourse cues shape naturalistic model behavior, with implications for improving conversational quality.

Abstract

Evaluating the naturalness of dialogue in language models (LMs) is not trivial: notions of 'naturalness' vary, and scalable quantitative metrics remain limited. This study leverages the linguistic notion of 'at-issueness' to assess dialogue naturalness and introduces a new method: Divide, Generate, Recombine, and Compare (DGRC). DGRC (i) divides a dialogue as a prompt, (ii) generates continuations for subparts using LMs, (iii) recombines the dialogue and continuations, and (iv) compares the likelihoods of the recombined sequences. This approach mitigates bias in linguistic analyses of LMs and enables systematic testing of discourse-sensitive behavior. Applying DGRC, we find that LMs prefer to continue dialogue on at-issue content, with this effect enhanced in instruct-tuned models. They also reduce their at-issue preference when relevant cues (e.g., "Hey, wait a minute") are present. Although instruct-tuning does not further amplify this modulation, the pattern reflects a hallmark of successful dialogue dynamics.
Paper Structure (19 sections, 4 equations, 5 figures, 3 tables)

This paper contains 19 sections, 4 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Dialogue naturalness and at-issueness
  3. Method
  4. Data, models, and implementation
  5. Experiments
  6. Experiment 1: At-issueness preference
  7. Results
  8. Experiment 2: Sensitivity to digression
  9. Results
  10. Conclusion
  11. Scope of construction
  12. At-issueness of constructions
  13. Understanding instruct-tuning
  14. Appendix
  15. Stimuli
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Visualization of the DGRC method, involving four steps: 1) Dividing the original utterance into sub-utterances; 2) Generating continuations for individual sub-utterances; 3) Recombining the sub-utterances into the original utterance, and 4) Comparing likelihoods for generated continuations. The end-result of this process allows us to characterize an LM's dialogue response dynamics. Note: We use log-probabilities per token in our analyses.
  • Figure 2: Experiment 1 results. VP2-preference of LMs (organized by parameter count) across arc and coord constructions, training mode (base/instruct), model family, and whether VP order was swapped.
  • Figure 3: Experiment 2 results. VP2-preference of LMs (organized by parameter count) across arc and coord constructions, training mode (base/instruct), model family, and digression signal---absent ("No, that's not true"); present ("Hey, wait a minute").
  • Figure 4: Effect of instruct-tuning on results in Experiment 1. VP2-preference of LMs across arc and coord structures, modulated by instruct-tuned vs. base model.
  • Figure 5: Effect of the digression signal in Experiment 2. VP2-preference of LMs by digression signal, modulated by arc and coord structures.