Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection

Jiaqi Li, Xinyi Dong, Yang Liu, Zhizhuo Yang, Quansen Wang, Xiaobo Wang, SongChun Zhu, Zixia Jia, Zilong Zheng

TL;DR

ReflectEvo introduces a pipeline for autonomously generating self-reflections and using them to train small language models to improve meta introspection and reasoning. A large-scale ReflectEvo-460k dataset of self-generated reflections across diverse domains enables multiple learning setups (SFT and DPO), yielding substantial gains over baselines and rivaling larger open-source models on BIG-bench without distillation. The approach demonstrates improved error localization, correction via iterative self-reflection, and evidence of cross-task and cross-model generalization, with deeper analyses of reflection types and correlations to performance. This work suggests a practical, scalable path for continuously enhancing SLM reasoning through self-evolution, while acknowledging limitations related to data quality, task specialization, and verifer design.

Abstract

We present a novel pipeline, ReflectEvo, to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning. This process iteratively generates self-reflection for self-training, fostering a continuous and self-evolving process. Leveraging this pipeline, we construct ReflectEvo-460k, a large-scale, comprehensive, self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. Building upon this dataset, we demonstrate the effectiveness of reflection learning to improve SLMs' reasoning abilities using SFT and DPO with remarkable performance, substantially boosting Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1%. It validates that ReflectEvo can rival or even surpass the reasoning capability of the three prominent open-sourced models on BIG-bench without distillation from superior models or fine-grained human annotation. We further conduct a deeper analysis of the high quality of self-generated reflections and their impact on error localization and correction. Our work highlights the potential of continuously enhancing the reasoning performance of SLMs through iterative reflection learning in the long run.

ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection

TL;DR

ReflectEvo introduces a pipeline for autonomously generating self-reflections and using them to train small language models to improve meta introspection and reasoning. A large-scale ReflectEvo-460k dataset of self-generated reflections across diverse domains enables multiple learning setups (SFT and DPO), yielding substantial gains over baselines and rivaling larger open-source models on BIG-bench without distillation. The approach demonstrates improved error localization, correction via iterative self-reflection, and evidence of cross-task and cross-model generalization, with deeper analyses of reflection types and correlations to performance. This work suggests a practical, scalable path for continuously enhancing SLM reasoning through self-evolution, while acknowledging limitations related to data quality, task specialization, and verifer design.

Abstract

We present a novel pipeline, ReflectEvo, to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning. This process iteratively generates self-reflection for self-training, fostering a continuous and self-evolving process. Leveraging this pipeline, we construct ReflectEvo-460k, a large-scale, comprehensive, self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. Building upon this dataset, we demonstrate the effectiveness of reflection learning to improve SLMs' reasoning abilities using SFT and DPO with remarkable performance, substantially boosting Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1%. It validates that ReflectEvo can rival or even surpass the reasoning capability of the three prominent open-sourced models on BIG-bench without distillation from superior models or fine-grained human annotation. We further conduct a deeper analysis of the high quality of self-generated reflections and their impact on error localization and correction. Our work highlights the potential of continuously enhancing the reasoning performance of SLMs through iterative reflection learning in the long run.

Paper Structure

This paper contains 36 sections, 10 equations, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. The ReflectEvo Generation Scheme
  3. Problem Definition and Prelinminary
  4. Reflection Generation
  5. Reflection Curation
  6. Reflection Learning on Self-Generated Data
  7. Reflection Learning
  8. Inference
  9. Experiments
  10. Performance Learning on ReflectEvo
  11. Overall Performance on Different Tasks
  12. Effect of Reflection from Teacher Model
  13. Scaling Multi-turn Self-reflection
  14. Generalization across Different Tasks and Models
  15. Effect of Different Verifiers on Self-Reflection
  16. ...and 21 more sections

Figures (8)

  • Figure 1: Overview pipeline of ReflectEvo.
  • Figure 2: (a) Task-dataset hierarchy distribution of ReflectEvo-460k. (b) Error type distribution of corrected thoughts identified by reflection in the test sets.
  • Figure 3: Performance training with ReflectEvo across different tasks on Llama-3-8B.
  • Figure 4: Performance in multi-turn self-reflection on Llama-3-8B after tuning.
  • Figure 5: Qualitative examples from the MATH. "False to True" and a "False to False" stand for successful and failed correction in the second turn respectively. The key snippets highlighted in green, red and yellow indicate correct, erroneous thought and reflection respectively.
  • ...and 3 more figures