Zero-Shot Commonsense Validation and Reasoning with Large Language Models: An Evaluation on SemEval-2020 Task 4 Dataset

TL;DR

This paper assesses zero-shot capabilities of large language models on SemEval-2020 Task 4, focusing on commonsense validation (Task A) and explanation (Task B). It benchmarks LLaMA3-70B, Gemma2-9B, and Mixtral-8x7B against fine-tuned transformers using zero-shot prompts on a 1,000-item test set per task. Results show LLaMA3-70B achieving 98.4% Task A accuracy while Task B explanations remain weaker at 93.4%, with Gemma2-9B at 97.9%/91.0%, and Mixtral underperforming (66.0%/50.9%), with output-format issues contributing to lower Task B performance. The study highlights that zero-shot prompting can approach human-level validation but struggles with causal reasoning required for explanations, pointing to future work in retrieval-augmented prompting and knowledge-graph integration to improve explanation generation. Overall, the work provides a rigorous empirical benchmark of zero-shot commonsense reasoning and informs prompt design and model adaptation for practical deployment.

Abstract

This study evaluates the performance of Large Language Models (LLMs) on SemEval-2020 Task 4 dataset, focusing on commonsense validation and explanation. Our methodology involves evaluating multiple LLMs, including LLaMA3-70B, Gemma2-9B, and Mixtral-8x7B, using zero-shot prompting techniques. The models are tested on two tasks: Task A (Commonsense Validation), where models determine whether a statement aligns with commonsense knowledge, and Task B (Commonsense Explanation), where models identify the reasoning behind implausible statements. Performance is assessed based on accuracy, and results are compared to fine-tuned transformer-based models. The results indicate that larger models outperform previous models and perform closely to human evaluation for Task A, with LLaMA3-70B achieving the highest accuracy of 98.40% in Task A whereas, lagging behind previous models with 93.40% in Task B. However, while models effectively identify implausible statements, they face challenges in selecting the most relevant explanation, highlighting limitations in causal and inferential reasoning.

Figures (1)

• Figure 1: The architecture of the commonsense validation and reasoning with zero-shot prompting of LLMs.