Eliciting Causal Abilities in Large Language Models for Reasoning Tasks

TL;DR

The paper addresses the challenge of improving LLM-based reasoning by eliciting causal inference through prompting. It introduces Self-Causal Instruction Enhancement (SCIE), a pipeline that generates high-quality observational data, estimates causal effects of prompting instructions using LLMs, and produces enhanced instructions, with an Object-Relational (OR) module for reusing causal templates across tasks. Empirical results across diverse reasoning tasks show SCIE improves accuracy while reducing prompt-training cost and providing interpretable, proxy-feature–driven insights; OR further demonstrates cost-effective reusability and generalization of learned prompting patterns. The work highlights the potential of causal meta-prompting to guide instruction design and paves the way for robust, interpretable prompt optimization in real-world applications.

Abstract

Prompt optimization automatically refines prompting expressions, unlocking the full potential of LLMs in downstream tasks. However, current prompt optimization methods are costly to train and lack sufficient interpretability. This paper proposes enhancing LLMs' reasoning performance by eliciting their causal inference ability from prompting instructions to correct answers. Specifically, we introduce the Self-Causal Instruction Enhancement (SCIE) method, which enables LLMs to generate high-quality, low-quantity observational data, then estimates the causal effect based on these data, and ultimately generates instructions with the optimized causal effect. In SCIE, the instructions are treated as the treatment, and textual features are used to process natural language, establishing causal relationships through treatments between instructions and downstream tasks. Additionally, we propose applying Object-Relational (OR) principles, where the uncovered causal relationships are treated as the inheritable class across task objects, ensuring low-cost reusability. Extensive experiments demonstrate that our method effectively generates instructions that enhance reasoning performance with reduced training cost of prompts, leveraging interpretable textual features to provide actionable insights.

Figures (6)

• Figure 1: Illustrative examples demonstrating the purpose of our proposed (Object-Relational) Self-Causal Instruction Enhancement method.
• Figure 2: The overall process of (Object-Relational) Self-Causal Instruction Enhancement includes Data Generation, Causal Effect Estimation, Enhanced Instructions Generation, and optional OR Module.
• Figure 3: Probability density distributions of the proxy features "Structure" and "Specification of Detail Level" with "Directness" as the proxy treatment.
• Figure 4: The causal effect estimation with the LLM (asterisk points), T-Learner (blue circular points), and S-Learner (red triangle points). The X-axis represents the $n$ proxy features for each reasoning task (indicated by numerical ticks), used as treatments when calculating $\mathrm{ATE_i}$. The Y-axis represents the corresponding ATE values of treatments.
• Figure 5: (a) The reasoning accuracy with and without SCIE, using Plan-and-Solve and AgentInstruct as base instructions respectively. (b) The reasoning accuracy for objects inheriting causal relationships with aggregation(A) and generalization(G) OR relationships, respectively.