WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization

TL;DR

This paper scrutinizes how Winograd-based benchmarks assess common sense reasoning in large language models by introducing WinoWhat, a paraphrased parallel corpus of the WinoGrande validation set. It evaluates open-source LLM families across sizes, using a paraphrase-friendly evaluation setup and per-category analyses to uncover fine-grained weaknesses in common sense knowledge. Five knowledge categories (physical, social, numerical, spatial, temporal) enable targeted error analysis, while memorization and data contamination are systematically examined. The findings show that paraphrasing substantially reduces performance across models, suggesting that previous results on WinoGrande may overstate reasoning capabilities and highlight the need for artifact-aware evaluation and interpretability; the WinoWhat resource is released to support ongoing research.

Abstract

In this study, we take a closer look at how Winograd schema challenges can be used to evaluate common sense reasoning in LLMs. Specifically, we evaluate generative models of different sizes on the popular WinoGrande benchmark. We release WinoWhat, a new corpus, in which each instance of the WinoGrande validation set is paraphrased. Additionally, we evaluate the performance on the challenge across five common sense knowledge categories, giving more fine-grained insights on what types of knowledge are more challenging for LLMs. Surprisingly, all models perform significantly worse on WinoWhat, implying that LLM reasoning capabilities are overestimated on WinoGrande. To verify whether this is an effect of benchmark memorization, we match benchmark instances to LLM trainingdata and create two test-suites. We observe that memorization has a minimal effect on model performance on WinoGrande.

Figures (4)

• Figure 1: Illustration of the workflow in this study. We evaluate LLMs on WinoGrande, and on its paraphrased variant. We further compare performance per common sense knowledge category, and check for benchmark memorization.
• Figure 2: An illustration of the paraphrasing and evaluation method. The option that is filled in the '_'-token is in red. In the original example, the summed log-likelihood is calculated on the tokens after the option. In our paraphrased corpus, the option is at the end of the sentence, and the summed log-likelihood is calculated on the tokens inside the option.
• Figure 3: Data distribution across common sense categories on the WinoGrande validation set.
• Figure 4: Scatter plots showing correlation for all instances in the WinoGrande validation set against the logit difference between ground truth correct and incorrect answers. (a) Correlation with n-gram length. (b) Correlation with n-gram count.