Estimating Commonsense Plausibility through Semantic Shifts

TL;DR

ComPaSS addresses the challenge of fine-grained commonsense plausibility evaluation by employing a discriminative, zero-shot approach that measures semantic shifts when augmenting sentences with commonsense cues. Anchors and candidate augmentations are encoded into semantic representations, and plausibility is inferred from the similarity between anchor and candidate embeddings, with $p^c_i \propto \mathrm{sim}(r_{anchor}, r_{candi})$. Across structured attribute and free-form general-knowledge tasks, ComPaSS consistently outperforms likelihood- and verbalization-based baselines and benefits substantially from contrastive pre-training and multimodal grounding, particularly on vision-grounded tasks. The method is architecture-agnostic, compatible with encoders and decoders, and demonstrates that discriminative, representation-level semantics can yield more precise fine-grained plausibility judgments, with practical implications for evaluating and guiding LM outputs. Overall, ComPaSS provides a scalable, backbone-agnostic framework for CSPE that leverages semantic embeddings to capture nuanced commonsense distinctions, enabling more reliable evaluation and augmentation of language systems.

Abstract

Commonsense plausibility estimation is critical for evaluating language models (LMs), yet existing generative approaches--reliant on likelihoods or verbalized judgments--struggle with fine-grained discrimination. In this paper, we propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts when augmenting sentences with commonsense-related information. Plausible augmentations induce minimal shifts in semantics, while implausible ones result in substantial deviations. Evaluations on two types of fine-grained commonsense plausibility estimation tasks across different backbones, including LLMs and vision-language models (VLMs), show that ComPaSS consistently outperforms baselines. It demonstrates the advantage of discriminative approaches over generative methods in fine-grained commonsense plausibility evaluation. Experiments also show that (1) VLMs yield superior performance to LMs, when integrated with ComPaSS, on vision-grounded commonsense tasks. (2) contrastive pre-training sharpens backbone models' ability to capture semantic nuances, thereby further enhancing ComPaSS.

Figures (6)

• Figure 1: How ComPaSS works on different tasks.
• Figure 2: Binary classification accuracy of models with ComPaSS on different groups.
• Figure 3: ComPaSS performance with different template types and template ensemble settings.
• Figure 4: The ranking of sheep colors by humans and different models, along with corresponding images from the physical world (from Google). The '*' in the upper right represents the model with ComPaSS method.
• Figure 5: The prompt for converting question-answer pair into sentence. The blue part is the instruction, the green part is the 3-shot example, and the red part is the placeholder for the specific input.