ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization

TL;DR

This work tackles the high cost of aligning large language models with human preferences by introducing ICDPO, a tuning-free approach that borrows alignment capabilities from superior models through in-context learning. It rethinks Direct Preference Optimization, derives an instant scorer from the states of the model before and after ICL, and employs a two-stage retrieval to curate demonstrations. A contrastive score guides candidate ranking without parameter updates, enabling the base LLM to align more closely with human preferences while remaining cost-efficient. Across multiple baselines and datasets, ICDPO demonstrates strong improvements over fine-tuning-free methods and competitive performance with SFT plus LoRA, with analyses highlighting the importance of demonstration quality, retrieval strategy, and scoring reliability.

Abstract

Large Language Models (LLMs) rely on Human Preference Alignment (HPA) to ensure the generation of safe content. Due to the heavy cost associated with fine-tuning, fine-tuning-free methods have emerged, typically modifying LLM decoding with external auxiliary methods. However, these methods do not essentially enhance the LLM itself. In this paper, we rethink the derivation procedures of DPO, based on which we conversely build an instant scorer using the states of the LLM before and after In-context Learning (ICL). Accordingly, we propose a novel approach called In-Context Direct Preference Optimization (ICDPO). It enables LLMs to borrow the HPA capabilities from superior LLMs with ICL, generating well-aligned responses as estimated by the aforementioned instant scorer, thereby enhancing the final performance. ICDPO can be further enhanced with a two-stage retriever and an upgraded scorer, both offering benefits. Extensive experiments show its effectiveness, particularly in outperforming two fine-tuning-free baselines, and it exhibits competitiveness with SFT + LoRA. We also conduct detailed analyses to offer comprehensive insights into ICDPO.

Figures (7)

• Figure 1: The overview of ICDPO. (a) The difference in teacher data utilization between normal fine-tuning and ICL without fine-tuning. (b) The core of ICDPO is that expert-amateur coordination maximizes $S$ which represents the disparity between the expert and the amateur. It brings more accurate estimation than using only the expert LLM.
• Figure 2: GPT-4 computed win-rates of ICDPO against golden responses in HH-RLHF, using demonstrations from the teacher (i.e. LLaMA2-chat). For each block titled by one base model, the bars from top to bottom are ICDPO, ICDPO$+\hat{S}$ and ICDPO$+\hat{S}R$, while red, light green and purple represent the proportion of win, tie and lose, respectively.
• Figure 3: Results of consistency between different scorers and GPT-4. We compute MRR to measure the degree of consistency. (a) Results with randomly selected demonstrations. (b) Results with demonstrations retrieved by $R$.
• Figure 4: GPT-4 computed win-rates of ICDPO against golden responses in HH-RLHF, using demonstrations from the teacher (i.e. GPT-3.5-turbo). For each block titled by one base model, the bars from top to bottom are ICDPO, ICDPO$+\hat{S}$ and ICDPO$+\hat{S}R$, while red, light green and purple represent the proportion of win, tie and lose, respectively.
• Figure 5: Loss of different base models on demonstrations from LLaMA2-chat and GPT-3.5-turbo.