Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Commonsense in Pre-trained Language Models

Xuhui Zhou, Yue Zhang, Leyang Cui, Dandan Huang

TL;DR

This paper benchmarks commonsense knowledge in pretrained contextualized language models by evaluating GPT, GPT2, BERT, XLNet, and RoBERTa on a unified CATs suite spanning seven tasks (SM, WSC, CA, SWAG, HellaSwag, SMR, ARCT). It reframes diverse datasets into sentence-scoring problems to systematically probe world knowledge, reasoning, and abductive abilities. Key findings show that bidirectional, larger-scale training improves commonsense performance, yet higher inference demands expose substantive gaps relative to human ability, and robustness tests reveal surface-level rather than deep commonsense understanding. The CATs benchmark and test scripts are released publicly to catalyze future research in robust commonsense reasoning.

Abstract

Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.

Evaluating Commonsense in Pre-trained Language Models

TL;DR

This paper benchmarks commonsense knowledge in pretrained contextualized language models by evaluating GPT, GPT2, BERT, XLNet, and RoBERTa on a unified CATs suite spanning seven tasks (SM, WSC, CA, SWAG, HellaSwag, SMR, ARCT). It reframes diverse datasets into sentence-scoring problems to systematically probe world knowledge, reasoning, and abductive abilities. Key findings show that bidirectional, larger-scale training improves commonsense performance, yet higher inference demands expose substantive gaps relative to human ability, and robustness tests reveal surface-level rather than deep commonsense understanding. The CATs benchmark and test scripts are released publicly to catalyze future research in robust commonsense reasoning.

Abstract

Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.

Paper Structure

This paper contains 19 sections, 5 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Tasks for Evaluating Commonsense
  3. Sense Making (SM)
  4. Winograd Schema Challenge (WSC)
  5. Conjunction Acceptability (CA)
  6. SWAG
  7. HellaSwag
  8. Sense Making with Reasoning (SMR)
  9. Argument Reasoning Comprehension Task (ARCT)
  10. Pre-trained Models
  11. Experimental Design
  12. Commonsense Tests Results
  13. Uni-directional Vs Bi-directional LM
  14. Scale of Training Data
  15. Number of Inference Steps
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Comparison between bidirectional and unidirectional models among different tasks.
  • Figure 2: Models performances when the number of inference step (IS) increases. Tasks are ranked according to their IS in an increasing order from left to right.
  • Figure 3: Samples of questions from Add, Del, Sub and Swap predicted correctly for the original instance but incorrectly for the dual instance. Note that a sentence here represents a test instance with a pair of positive and negative samples, represented by (.../...). Here we mark the correct prediction by an asterisk and display the normalized $q_t$ by coloring its corresponding word.