Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution

Ian Porada, Jackie Chi Kit Cheung

TL;DR

It is demonstrated that despite the strong performance of prompted language models on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier.

Abstract

Challenge sets such as the Winograd Schema Challenge (WSC) are used to benchmark systems' ability to resolve ambiguities in natural language. If one assumes as in existing work that solving a given challenge set is at least as difficult as solving some more general task, then high performance on the challenge set should indicate high performance on the general task overall. However, we show empirically that this assumption of difficulty does not always hold. In particular, we demonstrate that despite the strong performance of prompted language models (LMs) on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier. Motivated by these findings, we propose a method for ensembling a prompted LM with a supervised, task-specific system that is overall more accurate at resolving pronominal coreference across datasets. Finally, we emphasize that datasets involving the same linguistic phenomenon draw on distinct, but overlapping, capabilities, and evaluating on any one dataset alone does not provide a complete picture of a system's overall capability.

Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution

TL;DR

It is demonstrated that despite the strong performance of prompted language models on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier.

Abstract

Challenge sets such as the Winograd Schema Challenge (WSC) are used to benchmark systems' ability to resolve ambiguities in natural language. If one assumes as in existing work that solving a given challenge set is at least as difficult as solving some more general task, then high performance on the challenge set should indicate high performance on the general task overall. However, we show empirically that this assumption of difficulty does not always hold. In particular, we demonstrate that despite the strong performance of prompted language models (LMs) on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier. Motivated by these findings, we propose a method for ensembling a prompted LM with a supervised, task-specific system that is overall more accurate at resolving pronominal coreference across datasets. Finally, we emphasize that datasets involving the same linguistic phenomenon draw on distinct, but overlapping, capabilities, and evaluating on any one dataset alone does not provide a complete picture of a system's overall capability.

Paper Structure

This paper contains 74 sections, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Method
  5. Problem Formulation
  6. Challenge Set Assumptions
  7. Evaluating Performance
  8. Attested Pronominal Expressions.
  9. WSC-like Challenge Sets.
  10. Experiments
  11. Systems
  12. Prompted Language Models
  13. Llama 3.1
  14. Additional LMs
  15. Prompting Techniques.
  16. ...and 59 more sections

Figures (15)

  • Figure 1: Top: An example minimal pair from the WSC. Bottom: Pronouns attested in the novel Pride and Prejudice and annotated for coreference by vala-etal-2016-annotating.
  • Figure 2: An example instance and the corresponding variables: the pronoun $x$, antecedent $a$, and distractor candidate $b$.
  • Figure 3: A training set instance from the Definite Pronoun Resolution (DPR) dataset rahman-ng-2012-resolving formatted using each of the corresponding prompts. Denoted in bold is the expected model output. The GPT-3 prompt brown2020language does not rely on gold mention span annotations. QA Prompt and Doc Prompt were presented by le2023large. The multiple-choice QA (MQA) prompt was presented by zhu-etal-2024-large.
  • Figure 4: A comparison of the rule-based dcoref system lee-etal-2013-deterministic and the Llama 3.1 8B base model prompted for PCR using various prompts. A) Systems that do not need gold mention spans. Across datasets, Llama 3.1 with the GPT-3 prompt always outperforms the dcoref baseline. B) Systems that require gold mention spans as input. In general, prompted Llama 3.1 is more accurate than dcoref on both attested and constructed instances.
  • Figure 5: Accuracies of various LMs using the QA prompt template as compared against a dcoref baseline. We find that LMs generally outperform dcoref on both attested and constructed instances.
  • ...and 10 more figures

Theorems & Definitions (1)

  • definition 1: The Challenge Set Assumption