Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NEFTune: Noisy Embeddings Improve Instruction Finetuning

Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, Tom Goldstein

TL;DR

NEFTune presents a simple yet powerful regularization technique for instruction fine-tuning by adding random embedding noise during training. This non-adversarial perturbation substantially improves conversational quality, achieving large gains on AlpacaEval and consistent improvements across diverse instruction datasets, including Evol-Instruct, ShareGPT, and OpenPlatypus; it also benefits RLHF-refined chat models and remains effective under QLORA. Analyses indicate reduced overfitting and longer, more detailed outputs without sacrificing text diversity or core capabilities. Human studies corroborate increased perceived quality, underscoring NEFTune's practical impact as a low-cost enhancement for instruction-tuning pipelines.

Abstract

We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29.79% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instruct see a 10% improvement, with ShareGPT an 8% improvement, and with OpenPlatypus an 8% improvement. Even powerful models further refined with RLHF such as LLaMA-2-Chat benefit from additional training with NEFTune.

NEFTune: Noisy Embeddings Improve Instruction Finetuning

TL;DR

NEFTune presents a simple yet powerful regularization technique for instruction fine-tuning by adding random embedding noise during training. This non-adversarial perturbation substantially improves conversational quality, achieving large gains on AlpacaEval and consistent improvements across diverse instruction datasets, including Evol-Instruct, ShareGPT, and OpenPlatypus; it also benefits RLHF-refined chat models and remains effective under QLORA. Analyses indicate reduced overfitting and longer, more detailed outputs without sacrificing text diversity or core capabilities. Human studies corroborate increased perceived quality, underscoring NEFTune's practical impact as a low-cost enhancement for instruction-tuning pipelines.

Abstract

We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29.79% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instruct see a 10% improvement, with ShareGPT an 8% improvement, and with OpenPlatypus an 8% improvement. Even powerful models further refined with RLHF such as LLaMA-2-Chat benefit from additional training with NEFTune.
Paper Structure (29 sections, 9 figures, 14 tables, 1 algorithm)

This paper contains 29 sections, 9 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. NEFTune: Noisy Embedding Instruction Finetuning
  4. Experimental Set-up
  5. Models
  6. Instruction Finetuning Datasets
  7. Evaluation
  8. Results
  9. NEFTune Improves Text Quality.
  10. NEFTune Can Improve Chat Models.
  11. Effect on Capabilities.
  12. NEFTune Works with QLORA.
  13. A Qualitative Example.
  14. Analysis
  15. Overfitting
  16. ...and 14 more sections

Figures (9)

  • Figure 1: AlpacaEval Win Rate percentage for LLaMA-2-7B models finetuned on various datasets with and without NEFTune. NEFTune leads to massive performance boosts across all of these datasets, showcasing the increased conversational quality of the generated answers.
  • Figure 2: AlpacaEval Win Rate with and without NEFTune on LLaMA-2, LLaMA-1, and OPT across Alpaca, Evol-Instruct, ShareGPT and OpenPlatypus datasets. Performance improves across different datasets and models with ChatGPT as the evaluator.
  • Figure 3: OpenLLM Leaderboard tasks with and without NEFTune on LLaMA-2 across Alpaca, Evol-Instruct, and OpenPlatypus datasets and LLaMA-1 trained on Evol-Instruct. We observe that performance does not change across datasets and models.
  • Figure 4: Left: training loss on the Alpaca dataset for models with and without NEFT, computed with no added noise. Training with NEFT yields a higher training loss. Right: loss of the same model, but evaluated on the "test" Evol-Instruct dataset. NEFT yields slightly lower loss.
  • Figure 5: Left shows the ROUGE-L of training with and without NEFT. Right shows BLEU score.
  • ...and 4 more figures