Counterfactual Samples Constructing and Training for Commonsense Statements Estimation

TL;DR

The paper addresses weaknesses in Plausibility Estimation by introducing Commonsense Counterfactual Samples Generating (CCSG), a model-agnostic method that enforces language-explainable and commonsense-sensitive reasoning. CCSG combines a Counterfactual Samples Constructor (word-level substitutions guided by contribution and dropout-based augmentation) with a sentence-level contrastive training objective, underpinned by a Structural Causal Model (SCM) to mitigate commonsense bias. Evaluations across nine datasets show CCSG achieves a 3.07% absolute improvement over prior state-of-the-art and enhances LLM-based knowledge filtering, while ablations demonstrate improvements in bias reduction and explainability. The approach advances PE robustness, interpretability, and reliability, though it recognizes limitations in fictional contexts and ethical considerations for deployment.

Abstract

Plausibility Estimation (PE) plays a crucial role for enabling language models to objectively comprehend the real world. While large language models (LLMs) demonstrate remarkable capabilities in PE tasks but sometimes produce trivial commonsense errors due to the complexity of commonsense knowledge. They lack two key traits of an ideal PE model: a) Language-explainable: relying on critical word segments for decisions, and b) Commonsense-sensitive: detecting subtle linguistic variations in commonsense. To address these issues, we propose a novel model-agnostic method, referred to as Commonsense Counterfactual Samples Generating (CCSG). By training PE models with CCSG, we encourage them to focus on critical words, thereby enhancing both their language-explainable and commonsense-sensitive capabilities. Specifically, CCSG generates counterfactual samples by strategically replacing key words and introducing low-level dropout within sentences. These counterfactual samples are then incorporated into a sentence-level contrastive training framework to further enhance the model's learning process. Experimental results across nine diverse datasets demonstrate the effectiveness of CCSG in addressing commonsense reasoning challenges, with our CCSG method showing 3.07% improvement against the SOTA methods.

Figures (4)

• Figure 1: An ideal PE model should exhibit two indispensable characteristics: (a) Language-Explainable ability: The PE model should not only make a correct prediction but also base its prediction on the appropriate linguistic reference regions. (b) Commonsense-Sensitive ability: The PE model must be enough sensitive to commonsense variations in texts. For example, replacing the critical word "swim" with "run", in a statement should result in opposite predicted labels, reflecting the underlying change in plausibility.
• Figure 2: The SCM depicts the inferential mechanism of the PE model without any interventions. (b) An intervention on the variable $C$ is assigned the value $c_0$, represented as do($C = c_0$). Similarly, an intervention on the variable $K$ is assigned the value $k_0$, represented as do($K = k_0$)
• Figure 3: The overview of our CCSG framework is as follows: 1) We prepare appropriate keywords based on the word contributions from the previous training output. 2) We replace the keywords using a vector knowledge base (VKB), such as Glove, to generate counterfactual negative samples, and generate counterfactual positive samples by applying dropout gao2021simcse to the original samples. 3) We send the counterfactual samples to the PE model, where sentence-level contrastive learning is used to enhance the model’s commonsense reasoning ability.
• Figure 4: Visualization of the contribution distribution of each token.