Natural Language Fine-Tuning

TL;DR

This work addresses data-scarce domain-specific fine-tuning for large language models by introducing Natural Language Fine-Tuning (NLFT), a token-level method that uses natural language as supervisory signal to identify saliency tokens via conditional-probability contrasts under different prompts. The approach yields a token-weighted loss that focuses learning on the most informative parts of reasoning outputs, achieving strong GSM8K performance with minimal data (e.g., 64.29% accuracy with 50 samples) and no warm-up, while maintaining $O(n)$ time/space complexity. Empirically, NLFT outperforms supervised fine-tuning (SFT) and reinforcement-based fine-tuning (ReFT) in both accuracy and efficiency, with notably lower memory (≈46.9 GB) and faster per-epoch times than ReFT, and substantially tighter resources than SFT when data are scarce. The method offers a lightweight, interpretable path for edge deployment and promises broad applicability to reasoning tasks across coding, medical diagnosis, and natural language inference, where labeled data are limited and compute is constrained.

Abstract

Large language model fine-tuning techniques typically depend on extensive labeled data, external guidance, and feedback, such as human alignment, scalar rewards, and demonstration. However, in practical application, the scarcity of specific knowledge poses unprecedented challenges to existing fine-tuning techniques. In this paper, focusing on fine-tuning tasks in specific domains with limited data, we introduce Natural Language Fine-Tuning (NLFT), which utilizes natural language for fine-tuning for the first time. By leveraging the strong language comprehension capability of the target LM, NLFT attaches the guidance of natural language to the token-level outputs. Then, saliency tokens are identified with calculated probabilities. Since linguistic information is effectively utilized in NLFT, our proposed method significantly reduces training costs. It markedly enhances training efficiency, comprehensively outperforming reinforcement fine-tuning algorithms in accuracy, time-saving, and resource conservation. Additionally, on the macro level, NLFT can be viewed as a token-level fine-grained optimization of SFT, thereby efficiently replacing the SFT process without the need for warm-up (as opposed to ReFT requiring multiple rounds of warm-up with SFT). Compared to SFT, NLFT does not increase the algorithmic complexity, maintaining O(n). Extensive experiments on the GSM8K dataset demonstrate that NLFT, with only 50 data instances, achieves an accuracy increase that exceeds SFT by 219%. Compared to ReFT, the time complexity and space complexity of NLFT are reduced by 78.27% and 92.24%, respectively. The superior technique of NLFT is paving the way for the deployment of various innovative LLM fine-tuning applications when resources are limited at network edges. Our code has been released at https://github.com/Julia-LiuJ/NLFT.

Figures (10)

• Figure 1: Accuracy Comparison of SFT and NLFT on GSM8K dataset. NLFT has the same time and space complexity as SFT but achieves a 27% increase in fine-tuning performance, maintaining a stable performance advantage thereafter. With minimal dataset samples (only 50 data points), a brief training period (3 epochs, 287 seconds), and low computational resource consumption (44.46 GB of GPU memory usage), NLFT does not require a warm-up phase and can achieve a performance 1.19 times greater than SFT. According to the ReFT paper, ReFT is unable to outperform SFT within 8 epochs.
• Figure 2: Training process of SFT, ReFT, and NLFT. (a) The training process of SFT, which can be analogous to a student directly learning from a collection of exercises, which include problems and their reference solutions; (b) The training process of ReFT, which can be realized as a student repeatedly submitting their exam answers to a grading system, receiving scores, and striving to discover the strategies that maximize their marks; (c) The training process of our proposed algorithm NLFT, which is similar to a student submitting exam answers and receiving detailed feedback from a natural language evaluator. The system highlights the scoring points and losing points, allowing the student to learn from both well-graded examples (i.e., for learning from teaching) and their work (i.e., for self-study), thereby increasing their performance.
• Figure 3: An example of the training process of NLFT, which takes question, standard answer, and judgment as inputs and generates different Input Prompts. Then, under different prompts, the algorithm compares different conditional probabilities to allocate the saliency token and assign scale values.
• Figure 4: Comparison of accuracy of SFT and NLFT using 25%, 50%, 75%, and 100% of GSM8K training set, corresponding to 1868, 3737, 5605, and 7473 samples, respectively. At proportion of 0 represents base model before fine-tuning.
• Figure 5: Comparison of accuracy of NLFT using minimal dataset samples of GSM8K as a training set, including NLFT trained with 200 steps, 1 epoch, and 2 epochs, respectively. To better illustrate the increase in accuracy from the data size of 50 to 100, we additionally provide plots for data size of 75, under the settings of 1-epoch and 2-epoch training.