Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Validity of Coreference-based Evaluations of Natural Language Understanding

Ian Porada

TL;DR

This work critically examines coreference-based evaluations of natural language understanding, showing that many canonical benchmarks suffer from measurement validity issues due to contested definitions and divergent cross-dataset results. By introducing a novel event-plausibility evaluation, the author demonstrates that large language models achieve strong benchmark performance yet struggle with generalization and conceptual consistency. A controlled re-evaluation of coreference systems reveals that improvements often attributed to architecture are largely explained by the underlying language model, highlighting discriminant validity concerns. The thesis ultimately advocates for measurement-aware evaluation practices—including disaggregated analyses, cross-dataset validation, and explicit construct definitions—to drive more robust, generalizable NLP systems that better reflect human-like understanding.

Abstract

In this thesis, I refine our understanding as to what conclusions we can reach from coreference-based evaluations by expanding existing evaluation practices and considering the extent to which evaluation results are either converging or conflicting. First, I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions due to issues of measurement validity - including contestedness (multiple, competing definitions of coreference) and convergent validity (evaluation results that rank models differently across benchmarks). Second, I propose and implement a novel evaluation focused on testing systems' ability to infer the relative plausibility of events, a key aspect of resolving coreference. Through this extended evaluation, I find that contemporary language models demonstrate strong performance on standard benchmarks - improving over earlier baseline systems within certain domains and types of coreference - but remain sensitive to the evaluation conditions: they often fail to generalize in ways one would expect a human to be capable of when evaluation contexts are slightly modified. Taken together, these findings clarify both the strengths, such as improved accuracy over baselines on widely used evaluations, and the limitations of the current NLP paradigm, including weaknesses in measurement validity, and suggest directions for future work in developing better evaluation methods and more genuinely generalizable systems.

The Validity of Coreference-based Evaluations of Natural Language Understanding

TL;DR

This work critically examines coreference-based evaluations of natural language understanding, showing that many canonical benchmarks suffer from measurement validity issues due to contested definitions and divergent cross-dataset results. By introducing a novel event-plausibility evaluation, the author demonstrates that large language models achieve strong benchmark performance yet struggle with generalization and conceptual consistency. A controlled re-evaluation of coreference systems reveals that improvements often attributed to architecture are largely explained by the underlying language model, highlighting discriminant validity concerns. The thesis ultimately advocates for measurement-aware evaluation practices—including disaggregated analyses, cross-dataset validation, and explicit construct definitions—to drive more robust, generalizable NLP systems that better reflect human-like understanding.

Abstract

In this thesis, I refine our understanding as to what conclusions we can reach from coreference-based evaluations by expanding existing evaluation practices and considering the extent to which evaluation results are either converging or conflicting. First, I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions due to issues of measurement validity - including contestedness (multiple, competing definitions of coreference) and convergent validity (evaluation results that rank models differently across benchmarks). Second, I propose and implement a novel evaluation focused on testing systems' ability to infer the relative plausibility of events, a key aspect of resolving coreference. Through this extended evaluation, I find that contemporary language models demonstrate strong performance on standard benchmarks - improving over earlier baseline systems within certain domains and types of coreference - but remain sensitive to the evaluation conditions: they often fail to generalize in ways one would expect a human to be capable of when evaluation contexts are slightly modified. Taken together, these findings clarify both the strengths, such as improved accuracy over baselines on widely used evaluations, and the limitations of the current NLP paradigm, including weaknesses in measurement validity, and suggest directions for future work in developing better evaluation methods and more genuinely generalizable systems.
Paper Structure (201 sections, 17 equations, 33 figures, 15 tables)

This paper contains 201 sections, 17 equations, 33 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Point 1: Existing coreference-based evaluations are limited in the generality of the conclusions they allow one to reach.
  3. Point 2: Restricted coreference-based evaluations indicate that language models may be limited as systems of inferring representations of semantics.
  4. Uniting Point 1 and Point 2.
  5. Overview
  6. Chapter \ref{['chap:challenge-set-assumption']}
  7. Chapter \ref{['chap:measurement']}
  8. Chapter \ref{['chap:supervised-comparison']}
  9. Chapter \ref{['chap:semantic-plausibility']}
  10. Discussion and Conclusion
  11. Literature Review
  12. Coreference Resolution
  13. Coreference-based Evaluations
  14. The Winograd Schema Challenge.
  15. Evaluation Metrics
  16. ...and 186 more sections

Figures (33)

  • Figure 1: The evolution of cuneiform written characters. Reproduced from maspero1916recueil (Creative Commons Attribution-Share Alike 4.0 International license).
  • Figure 2: Demonstrative examples of coreferences annotated in two popular datasets: OntoNotes marcus2011ontonotes and Winogrande winogrande_2020. Square brackets mark coreferring linguistic expressions, and subscripts indicate the entities to which the expressions refer.
  • Figure 3: A visualized outline of the structure of this thesis. The body of the thesis focuses on empirical experiments. Certain experiments concern either a) canonical corpus evaluations or b) targeted evaluations of an important aspect of resolving coreference, inferring event plausibility, as outlined in Section \ref{['sec:structure-overview']}.
  • Figure 4: The SHRDLU program proposed by Terry Winograd which was designed to interact with a virtual world of 3D objects based on natural language instructions. On the left is a visualization of the virtual world of objects, and on the right an example interaction between a person and the program. Figure reproduced from winograd_1972. (M.I.T. Project MAC, Defense Technical Information Center, Public Domain)
  • Figure 5: An illustration of how model performance is most commonly evaluated using the Winogrande dataset winogrande_2020 as an example. For each instance in the dataset, predicted coreferences are elicited from the model being evaluated and compared against coreferences annotated by a human annotator. Accuracy is then calculated across all examples in the test set. (Certain details regarding data formatting are simplified for illustrative purposes. The spelling of "beginner" is as this example appears in the original dataset.)
  • ...and 28 more figures

Theorems & Definitions (1)

  • Definition 1: The Challenge Set Assumption