Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoning

TL;DR

reflective augmentation is proposed, a method that embeds problem reflection into each training instance, thereby fostering a thorough comprehension through reflective reasoning and enhancing performance not only in standard settings but also in more complex scenarios that require reflective thinking.

Abstract

Supervised fine-tuning enhances the problem-solving abilities of language models across various mathematical reasoning tasks. To maximize such benefits, existing research focuses on broadening the training set with various data augmentation techniques, which is effective for standard single-round question-answering settings. Our work introduces a novel technique aimed at cultivating a deeper understanding of the training problems at hand, enhancing performance not only in standard settings but also in more complex scenarios that require reflective thinking. Specifically, we propose reflective augmentation, a method that embeds problem reflection into each training instance. It trains the model to consider alternative perspectives and engage with abstractions and analogies, thereby fostering a thorough comprehension through reflective reasoning. Extensive experiments validate the achievement of our aim, underscoring the unique advantages of our method and its complementary nature relative to existing augmentation techniques.

Figures (8)

• Figure 1: Question augmentation creates new questions based on existing ones. Answer augmentation re-samples answers for each problem to increase diversity. Both methods expand the size of the training set. Reflective augmentation appends the original answer with a reflective section, which is complementary to traditional approaches. Corresponding training sequences are shown in an (input, output) format, where augmented parts are in red.
• Figure 2: The model that learned the standard solution does not fully understand when and how to apply substitution when facing a different scenario. In contrast, the model trained with reflection on the substitution technique gains a deeper understanding of its principles, patterns, and its flexible application in new contexts.
• Figure 3: Relationship between the original instance and the reflective section. Either abstraction or analogy is annotated for each instance. Core ideas are shown but textual explanations (like those in Figure \ref{['fig:intro_example']}) are omitted.
• Figure 4: Average accuracy on 7 standard math reasoning tasks when different proportions of data are augmented with reflective sections (remaining data are in the standard QA form).
• Figure 5: Prompt used for training the model. Text in gray are placeholders and will be replaced by the corresponding sections in the training instance.