Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-Instruction-Free LLM Self-Alignment with Limited Samples

Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, Yang Liu

TL;DR

ISARA presents a data-efficient, human-instruction-free framework for aligning LLMs to target domains with limited seeds. It leverages retrieval-augmented in-context learning to generate high-quality self-labeled data and employs iterative supervised fine-tuning to progressively improve alignment. Across safety, truthfulness, and instruction-following benchmarks, ISARA demonstrates superior alignment, robust domain generalization, and favorable data-efficiency, even for smaller models. The approach eliminates the need for handcrafted instructions or reward models, enabling scalable, autonomous alignment with limited supervision.

Abstract

Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.

Human-Instruction-Free LLM Self-Alignment with Limited Samples

TL;DR

ISARA presents a data-efficient, human-instruction-free framework for aligning LLMs to target domains with limited seeds. It leverages retrieval-augmented in-context learning to generate high-quality self-labeled data and employs iterative supervised fine-tuning to progressively improve alignment. Across safety, truthfulness, and instruction-following benchmarks, ISARA demonstrates superior alignment, robust domain generalization, and favorable data-efficiency, even for smaller models. The approach eliminates the need for handcrafted instructions or reward models, enabling scalable, autonomous alignment with limited supervision.

Abstract

Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.
Paper Structure (34 sections, 4 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 34 sections, 4 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Model Alignment
  4. Retrieval-Augmented ICL
  5. Self-Alignment
  6. Preliminaries
  7. Methodology
  8. Data Generation
  9. Finetuning
  10. Iterative Enhancement
  11. Experiments
  12. Setup
  13. BeaverTails: Safety Alignment Experiment
  14. Performance.
  15. Ablation study on model sizes.
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Overview of ISARA. The only input is a few seed examples (e.g. < 100) from the target domain. We align the LLM iteratively, alternating between fine-tuning the LLM on self-generated samples (from retrieval-augmented ICL) and using the aligned LLM to generate new samples used to further align itself.
  • Figure 2: Domain generalization evaluation of ISARA on LLaMA-7B. For ISARA, iteration 0 corresponds to the pretrained model.
  • Figure 3: Utility evaluation for safety alignment.
  • Figure 4: Performance of ISARA in instruction-following alignment on LLaMA-7B and LLaMA-2-7B. We calculate the winning rate of ISARA against the other methods with GPT-4 as the judge following alpaca_eval.