ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Bases

TL;DR

ConstraintChecker addresses the difficulty of CSKB reasoning for LLMs by introducing a two-module plug-in that first derives explicit relational constraints from CSKB relations and then uses zero-shot checking with the same LLM to verify constraint satisfaction. The final decision is produced by conjunction of the main task output and the constraint-checking result, effectively reducing false positives. The authors demonstrate consistent, significant improvements over diverse prompting baselines on CKBPv2 and SD-ATOMIC20^20 across ChatGPT and GPT-3.5, supported by extensive ablations and analyses. This plug-in design offers a practical, cost-efficient enhancement to LLM-based CSKB reasoning with broad applicability and open-source access.

Abstract

Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has been explored as a way to acquire new commonsense knowledge based on reference knowledge in the original CSKBs and external prior knowledge. Despite the advancement of Large Language Models (LLM) and prompt engineering techniques in various reasoning tasks, they still struggle to deal with CSKB reasoning. One of the problems is that it is hard for them to acquire explicit relational constraints in CSKBs from only in-context exemplars, due to a lack of symbolic reasoning capabilities (Bengio et al., 2021). To this end, we proposed **ConstraintChecker**, a plugin over prompting techniques to provide and check explicit constraints. When considering a new knowledge instance, ConstraintChecker employs a rule-based module to produce a list of constraints, then it uses a zero-shot learning module to check whether this knowledge instance satisfies all constraints. The acquired constraint-checking result is then aggregated with the output of the main prompting technique to produce the final output. Experimental results on CSKB Reasoning benchmarks demonstrate the effectiveness of our method by bringing consistent improvements over all prompting methods. Codes and data are available at \url{https://github.com/HKUST-KnowComp/ConstraintChecker}.

Figures (2)

• Figure 1: Examples of CSKB Reasoning. Solid arrows represent existing triples (i.e., instances) in CSKBs, while the dashed arrows with question marks represent new knowledge triples which will be determined if they are commonsense. LLMs often fail to acquire the explicit relational constraints in CSKBs, hence making wrong predictions for many new knowledge triples.
• Figure 2: Illustration of ConstraintChecker. For each instance, Module 1 queries preset rules to get all relational constraints corresponding to the instance's relation. Module 2 then constructs questions accordingly to ask whether each constraint is satisfied, and passes these questions to the backbone LLM to get predictions. Together with the prediction from the main-task component, we use the logical conjunction (i.e., $\wedge$ or AND operator) to aggregate the final prediction. Note that "Prompt Engineering" refers to baseline prompting methods (subsection \ref{['sec:setup']}).