Debiasing Large Language Models via Adaptive Causal Prompting with Sketch-of-Thought

TL;DR

ACPS tackles the challenge of debiasing large language models while maintaining efficiency by integrating structural causal models with adaptive front-door prompting and Sketch-of-Thought. It replaces verbose CoT with concise SoT and uses a classification engine to choose between standard and conditional front-door interventions, guiding evidence-based answer selection without task-specific retraining. The framework estimates causal effects via reasoning-trace distribution, NWGM-based answer probabilities, and external-knowledge integration, demonstrating consistent gains across seven datasets and three backbone LLMs. This approach offers scalable, generalisable debiasing with improved accuracy, robustness, and token efficiency, advancing trustworthy reasoning in LLMs for diverse applications.

Abstract

Despite notable advancements in prompting methods for Large Language Models (LLMs), such as Chain-of-Thought (CoT), existing strategies still suffer from excessive token usage and limited generalisability across diverse reasoning tasks. To address these limitations, we propose an Adaptive Causal Prompting with Sketch-of-Thought (ACPS) framework, which leverages structural causal models to infer the causal effect of a query on its answer and adaptively select an appropriate intervention (i.e., standard front-door and conditional front-door adjustments). This design enables generalisable causal reasoning across heterogeneous tasks without task-specific retraining. By replacing verbose CoT with concise Sketch-of-Thought, ACPS enables efficient reasoning that significantly reduces token usage and inference cost. Extensive experiments on multiple reasoning benchmarks and LLMs demonstrate that ACPS consistently outperforms existing prompting baselines in terms of accuracy, robustness, and computational efficiency.

Figures (10)

• Figure 1: An example from GPT-3.5-turbo on the GSM8K dataset. Left: (i) Recent non-causal prompting methods often amplify internal bias through majority voting. (ii) Some causality-based prompting methods mitigate this bias but rely on verbose CoT, leading to high token usage and inference cost. Right: (iii) The proposed framework uses SoT instead of CoT and selects the answer based on the highest estimated causal effect, yielding the correct result.
• Figure 2: Three SCMs illustrate different modes of reasoning in LLMs: (a) direct prompt-to-answer reasoning; (b) causality-based prompting for tasks without external knowledge, such as Causal Prompting zhang2024causal; and (c) causality-based prompting for tasks with external knowledge. Both (b) and (c) are integrated into ACPS. In all SCMs, $Q$ denotes the query, $R$ denotes the reasoning process (SoT or CoT), $A$ denotes the answer, $U$ denotes the unobserved confounder, and $E$ denotes the external knowledge.
• Figure 3: Overall architecture of ACPS. Given an input $Q$ comprising the demonstration examples $[d_1, \ldots, d_n]$ and the test query $q$, a classification engine (CE) determines an appropriate intervention. The LLM generates $M$ diverse SoTs, which are embedded and clustered into $K$ groups. For each cluster representative, optimal demonstrations are selected via an encoder-based intervention algorithm to form updated prompts $\mathcal{P}^{\text{iter}}_{k}$. The LLM is then queried $S$ times per prompt, and the final answer is selected as the one associated with the highest estimated causal effect.
• Figure 4: Comparison of the average number of reasoning steps across all datasets for different prompting methods.
• Figure 5: Trends of (a) evaluation loss and (b) sts-dev_pearson_cosine during encoder fine-tuning on GSM8K.

Theorems & Definitions (3)

• Definition 1: Standard Front-Door Criterion
• Definition 2: Conditional Front-Door Criterion
• Theorem 1: Rules of $do$-Calculus Pearl2009Causality