Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving

TL;DR

LEPA tackles the problem of what information self-generated data should contain for LLM self-training by prompting models to generate anticipatory plans before solving problems; it alternates data-generation and supervised-fine-tuning phases, using self-reflection to refine plans and training to predict both refined plans and final solutions via the loss $\mathcal{L}_{SFT}$. It demonstrates superior performance on Hendrycks MATH, Hellaswag, BoolQ, and PIQA, outperforming ReST, STaR, and baseline CoT variants, with RL variants showing promise. The approach reduces cognitive load and captures transferable meta-knowledge, offering a scalable pathway to improved reasoning in LLMs and enabling future integration with RL and larger models.

Abstract

In the field of large language model (LLM) post-training, the effectiveness of utilizing synthetic data generated by the LLM itself has been well-presented. However, a key question remains unaddressed: what essential information should such self-generated data encapsulate? Existing approaches only produce step-by-step problem solutions, and fail to capture the abstract meta-knowledge necessary for generalization across similar problems. Drawing insights from cognitive science, where humans employ high-level abstraction to simplify complex problems before delving into specifics, we introduce a novel self-training algorithm: LEarning to Plan before Answering (LEPA). LEPA trains the LLM to formulate anticipatory plans, which serve as abstract meta-knowledge for problem-solving, before engaging with the intricacies of problems. This approach not only outlines the solution generation path but also shields the LLM from the distraction of irrelevant details. During data generation, LEPA first crafts an anticipatory plan based on the problem, and then generates a solution that aligns with both the plan and the problem. LEPA refines the plan through self-reflection, aiming to acquire plans that are instrumental in yielding correct solutions. During model optimization, the LLM is trained to predict both the refined plans and the corresponding solutions. By efficiently extracting and utilizing the anticipatory plans, LEPA demonstrates remarkable superiority over conventional algorithms on various challenging natural language reasoning benchmarks.

Figures (5)

• Figure 1: A didactic example demonstrating how LEPA outperforms baseline methods by learning to generate anticipatory plans before answering. (a) An example problem in the Hendrycks MATH test set. (b) An incorrect solution given by the LLM trained with a baseline method, ReST. The model fails to generate correct reasoning steps. (c) A correct solution given by the LLM trained with our proposed method, LEPA. The model generates high-quality plans, and then follows the plan to solve the problem correctly.
• Figure 2: Comparison between baseline algorithms' and LEPA's data generation procedure. (a) Baseline algorithms only generate step-by-step solutions to each problem, lacking high-level abstract meta-knowledge that guides solution generation. (b) LEPA generates anticipatory plans before generating detailed problem solutions. These plans are optimized with self-reflection, and encapsulate the high-level abstract problem-solving steps. The plans efficiently guide the LLM to generate correct solutions.
• Figure 3: Algorithms' learning curves on the four benchmarks. LEPA achieves better performance than baseline algorithms.
• Figure 4: A case study demonstrating how LEPA optimizes the anticipatory plans and the solutions with self-reflection. The initial plan is too broad and lacks detail, and fails to provide enough guidance to generate correct answers. The self-reflection process successfully analyses what is wrong, and generates a new, high-quality plan that provides more guidance while maintaining generalizability. With the new plan after self-reflection, the model successfully generates correct solutions.
• Figure 5: Detailed prompts used by (a) LEPA and (b) baseline algorithms.