HFT: Half Fine-Tuning for Large Language Models

TL;DR

Catastrophic forgetting during sequential fine-tuning of large language models is a critical bottleneck. The authors propose Half Fine-Tuning (HFT), freezing half of the parameters ($p=0.5$) and updating the rest, with a theoretical framing as masked optimization and a regularization term. Empirically, HFT alleviates forgetting across SFT, DPO, and continual learning, preserves basic knowledge while enabling new abilities, and yields substantial training-time savings (~$30 ext{ extpercent}$). The approach requires no architectural changes and is easily integrated into existing fine-tuning pipelines, offering a practical path for scalable continual tuning of large models.

Abstract

Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities, enabling LLMs to follow natural language instructions or align with human preferences. However, it carries the risk of catastrophic forgetting during sequential training, the parametric knowledge or the ability learned in previous stages may be overwhelmed by incoming training data. In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge. Inspired by this, we introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks while the other half are frozen to remain previous knowledge. We provide a feasibility analysis from the perspective of optimization and interpret the parameter selection operation as a regularization term. Without changing the model architecture, HFT could be seamlessly integrated into existing fine-tuning frameworks. Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT. Compared with FFT, HFT not only significantly alleviates the forgetting problem, but also achieves the best performance in a series of downstream benchmarks, with an approximately 30% reduction in training time.

Figures (8)

• Figure 1: Performance of Llama 2-7b, Llama 2-chat-7b, and the Half-Reset model on six general abilities and three basic knowledge benchmarks. It is interesting that simply resetting half of the parameters of the chat-model to the pre-trained model could roughly restore a significant amount of forgotten basic knowledge while maintaining high-level general abilities performance.
• Figure 2: The schematic procedure of HFT with Llama 2's architecture. In each stage, we selectively freeze half of the parameters at the category-level and update the other half. Best viewed in color.
• Figure 3: Performance of models trained on Tülu V2.
• Figure 4: Performance of models trained on TRACE.
• Figure 6: Variations on SAN in various transformer blocks.