LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

TL;DR

LOGICAL-COMMONSENSEQA addresses the limitations of single-answer commonsense benchmarks by modeling compositional plausibility with AND/OR/NEITHER/NOR over pairs of atomic statements. The authors construct a three-stage pipeline to convert CommonsenseQA into 19,996 four-option MCQs that expose joint plausibility relations while preserving the MCQ format. Empirical results show that language models handle conjunctive and disjunctive reasoning reasonably well but struggle with negation-based compositions, especially in zero- and few-shot settings; fine-tuning improves performance substantially, indicating the task is learnable with supervision. The benchmark provides a controlled framework for advancing compositional commonsense reasoning and is publicly available for broader evaluation and development.

Abstract

Commonsense reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most benchmarks rely on single-label evaluation, obscuring whether statements are jointly plausible, mutually exclusive, or jointly implausible. We introduce LOGICAL-COMMONSENSEQA, a benchmark that re-frames commonsense reasoning as logical composition over pairs of atomic statements using plausibility-level operators (AND, OR, NEITHER/NOR). Evaluating instruction-tuned, reasoning-specialized, and fine-tuned models under zero-shot, few-shot, and chain-of-thought prompting, we find that while models perform reasonably on conjunctive and moderately on disjunctive reasoning, performance degrades sharply on negation-based questions. LOGICAL-COMMONSENSEQA exposes fundamental reasoning limitations and provides a controlled framework for advancing compositional commonsense reasoning.