Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Retrieved In-Context Principles from Previous Mistakes

Hao Sun, Yong Jiang, Bo Wang, Yingyan Hou, Yan Zhang, Pengjun Xie, Fei Huang

TL;DR

RICP addresses limitations in error-driven prompting by introducing a retrieval-based, two-tiered principle framework that learns from the student’s mistakes. It employs a three-stage pipeline—Insight Generation, Principle Formulation, and Principle Utilization—where insights are clustered to form task-level principles and per-question retrieval yields customized question-level principles, appended to prompts during inference. Across seven benchmarks spanning mathematical, commonsense, and logical reasoning, RICP improves performance consistently across standard, zero-shot, and CoT prompting strategies, with notable gains when demonstrations are minimal. The method maintains no teacher intervention during inference and is compatible with existing prompting methods, offering scalable, targeted guidance to reduce repeated mistakes.

Abstract

In-context learning (ICL) has been instrumental in adapting Large Language Models (LLMs) to downstream tasks using correct input-output examples. Recent advances have attempted to improve model performance through principles derived from mistakes, yet these approaches suffer from lack of customization and inadequate error coverage. To address these limitations, we propose Retrieved In-Context Principles (RICP), a novel teacher-student framework. In RICP, the teacher model analyzes mistakes from the student model to generate reasons and insights for preventing similar mistakes. These mistakes are clustered based on their underlying reasons for developing task-level principles, enhancing the error coverage of principles. During inference, the most relevant mistakes for each question are retrieved to create question-level principles, improving the customization of the provided guidance. RICP is orthogonal to existing prompting methods and does not require intervention from the teacher model during inference. Experimental results across seven reasoning benchmarks reveal that RICP effectively enhances performance when applied to various prompting strategies.

Retrieved In-Context Principles from Previous Mistakes

TL;DR

RICP addresses limitations in error-driven prompting by introducing a retrieval-based, two-tiered principle framework that learns from the student’s mistakes. It employs a three-stage pipeline—Insight Generation, Principle Formulation, and Principle Utilization—where insights are clustered to form task-level principles and per-question retrieval yields customized question-level principles, appended to prompts during inference. Across seven benchmarks spanning mathematical, commonsense, and logical reasoning, RICP improves performance consistently across standard, zero-shot, and CoT prompting strategies, with notable gains when demonstrations are minimal. The method maintains no teacher intervention during inference and is compatible with existing prompting methods, offering scalable, targeted guidance to reduce repeated mistakes.

Abstract

In-context learning (ICL) has been instrumental in adapting Large Language Models (LLMs) to downstream tasks using correct input-output examples. Recent advances have attempted to improve model performance through principles derived from mistakes, yet these approaches suffer from lack of customization and inadequate error coverage. To address these limitations, we propose Retrieved In-Context Principles (RICP), a novel teacher-student framework. In RICP, the teacher model analyzes mistakes from the student model to generate reasons and insights for preventing similar mistakes. These mistakes are clustered based on their underlying reasons for developing task-level principles, enhancing the error coverage of principles. During inference, the most relevant mistakes for each question are retrieved to create question-level principles, improving the customization of the provided guidance. RICP is orthogonal to existing prompting methods and does not require intervention from the teacher model during inference. Experimental results across seven reasoning benchmarks reveal that RICP effectively enhances performance when applied to various prompting strategies.
Paper Structure (28 sections, 4 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 28 sections, 4 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Insight Generation
  5. Principle Formulation
  6. Task-level Principles
  7. Question-level Principles
  8. Principle Utilization
  9. Experiment
  10. Experiment Setup
  11. Baselines
  12. Reasoning Tasks
  13. Models
  14. Main Results
  15. Ablation Study
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Inference pipeline of RICP. The teacher model analyzes mistakes from the student model, creating guiding principles. These principles help prevent the student model from making similar mistakes.
  • Figure 2: The pipeline of RICP includes: 1) Insight Generation: The teacher model analyzes the student model's mistakes and generates high-level reasons and specific insights; 2) Principle Formulation: Task-level principle and question-level principles are generated based on the result of hierarchical clustering; 3) Principle Utilization: principles are integrated into the existing prompt to enhance the performance of the student model.
  • Figure 3: Ablation Study.
  • Figure 4: Hyper-parameters
  • Figure 5: The Comparison between Customized Retrieval and Random Selection.
  • ...and 2 more figures