Internalized Self-Correction for Large Language Models

TL;DR

This paper tackles the problem of superficial alignment in large language models trained with RLHF. It introduces Internalized Self-Correction (InSeC), a training-time mechanism that internalizes the correction process by pairing mistakes with their remedies and using negative sampling to provide a true supervised signal. Experiments on synthetic data with a Meta LLama3.1 8B model show that models trained with negative samples can auto-correct errors, whereas those trained without negatives do not. The work suggests that InSeC can improve sample efficiency, generalization, and instruction following, though it faces challenges like increased computation and careful negative-sample selection, and points to future integration with other feedback approaches.

Abstract

In this article, we introduce 'Internalized Self-Correction' (InSeC) for large language models (LLMs). While many approaches exist for self-reflection at inference time, we propose a novel method that combines ideas from negative sampling, self-reflection during training, and inference time. InSeC allows LLMs to correct themselves by introducing mistakes and their corresponding corrections during training, thereby converting the learning process into a true supervised learning task with both positive and negative examples. This approach can be extended to improve instruction following and correct hallucinations or incorrect sentences generated by LLMs.