Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

TL;DR

This work systematically investigates multi-round bootstrapping for self-alignment of LLMs and introduces Step-On-Feet Tuning (SOFT) to leverage a model’s improving few-shot abilities. Key innovations include expanding in-context learning example diversity, adopting an easy-to-hard training curriculum, and implementing a validation set to detect collapse, enabling more robust bootstrapping in early rounds. The authors also extend the approach with SOFT+, employing a perplexity-based curriculum to further enhance data quality and learning efficiency. Across classification and generation benchmarks, SOFT demonstrates superior performance over single-round methods and approaches specialized distilled models, highlighting the practical potential of continually refreshing self-alignment data. The study uncovers mechanisms behind late-stage degradation, such as data processing inequality and sharper output distributions, and offers practical strategies for mitigating collapse in real-world deployment.

Abstract

Self-alignment is an effective way to reduce the cost of human annotation while ensuring promising model capability. However, most current methods complete the data collection and training steps in a single round, which may overlook the continuously improving ability of self-aligned models. This gives rise to a key query: What if we do multi-time bootstrapping self-alignment? Does this strategy enhance model performance or lead to rapid degradation? In this paper, our pioneering exploration delves into the impact of bootstrapping self-alignment on large language models. Our findings reveal that bootstrapping self-alignment markedly surpasses the single-round approach, by guaranteeing data diversity from in-context learning. To further exploit the capabilities of bootstrapping, we investigate and adjust the training order of data, which yields improved performance of the model. Drawing on these findings, we propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance. Based on easy-to-hard training recipe, we propose SOFT+ which further boost self-alignment's performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across various classification and generation tasks, highlighting the potential of bootstrapping self-alignment on continually enhancing model alignment performance.

Figures (6)

• Figure 1: The workflow of SOFT. The model first takes in the combination of few shot demonstrations and task questions to generate high quality responses. The ICL examples used are randomly sampled in each batch. Then the responses are used to fine-tune the inference model. After this, the fine-tuned model will serve as the inference model to do the next round of inference.
• Figure 2: The figure demonstrates three round bootstrapping self-alignment evaluation on Truthful QA benchmark. The models are all evaluated one shot. It's obvious that bootstrapping aligned model better than the single-round method.
• Figure 3: Bootstrapping Self-Alignment vs Bootstrapping Self-Alignment from easy to hard. Three round self-alignment evaluation on Vicuna bench
• Figure 4: Bootstrapping Self-Alignment vs Bootstrapping Self-Alignment from easy to hard. Five round self-alignment evaluation on Vicuna bench
• Figure 5: Bootstrapping Self-Alignment vs Bootstrapping Self-Alignment from easy to hard. Seven round self-alignment evaluation on Vicuna bench