WSC+: Enhancing The Winograd Schema Challenge Using Tree-of-Experts

TL;DR

WSC+ introduces a generation-focused benchmark for Winograd-style coreference, augmented with Ambiguous and Offensive categories to probe model biases and overconfidence. The Tree-of-Experts (ToE) prompting framework significantly improves valid-instance generation, achieving $49.3\%$ validity (vs. $14.7\%$ for CoT) and enabling a large-scale dataset of $3{,}026$ validated questions. Across GPT-4, GPT-3.5, and Claude2, WSC+ reveals that even top LLMs struggle to reach human performance, with GPT-4 attaining $68.7\%$ accuracy on WSC+ compared to a $95.1\%$ human baseline, and exhibits notable biases and generation-evaluation inconsistencies. The work highlights fundamental gaps in reasoning fidelity and safety, offering directions for robust evaluation, bias mitigation, and improved prompt design in LLM-based dataset creation.

Abstract

The Winograd Schema Challenge (WSC) serves as a prominent benchmark for evaluating machine understanding. While Large Language Models (LLMs) excel at answering WSC questions, their ability to generate such questions remains less explored. In this work, we propose Tree-of-Experts (ToE), a novel prompting method which enhances the generation of WSC instances (50% valid cases vs. 10% in recent methods). Using this approach, we introduce WSC+, a novel dataset comprising 3,026 LLM-generated sentences. Notably, we extend the WSC framework by incorporating new 'ambiguous' and 'offensive' categories, providing a deeper insight into model overconfidence and bias. Our analysis reveals nuances in generation-evaluation consistency, suggesting that LLMs may not always outperform in evaluating their own generated questions when compared to those crafted by other models. On WSC+, GPT-4, the top-performing LLM, achieves an accuracy of 68.7%, significantly below the human benchmark of 95.1%.

Figures (16)

• Figure 1: Overview of the WSC+ generation and evaluation processes. On the left, the flowchart depicts the WSC+ generation process, using a real example generated by GPT-4. On the right, a WSC+ instance evaluation contrasts the outcomes of standard prompting and our Tree-of-Experts prompting.
• Figure 2: Percentage distribution of validity categories across LLMs in Winograd Schema sentence generation.
• Figure 3: Distribution of valid, semi-valid, and invalid WSC+ instances generated across various prompting strategies combined with query types.
• Figure 4: Comparison of pronoun distribution between the original WSC (WSC285) and WSC+ datasets.
• Figure 5: Performance of LLMs on the 100-pair subset of the WSC+ validation set with various prompting techniques.