Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model

TL;DR

The paper addresses cross-lingual alignment of foundation LLMs by questioning the depth of SFT-based alignment and proposing a training-free alternative called Pretty that uses minimal prior tokens to steer decoding. It analyzes the mechanisms behind SFT alignment, introducing the Prior Knowledge Hypothesis and showing that a single or two prior tokens can align decoding distributions between foundation and SFT models via $P_{ m LM}$ vs. $P_{ m SFT}$ similarities. Pretty constructs priors from SFT outputs, refined-task models, or dictionaries, and demonstrates competitive performance across machine translation, cross-lingual summarization, and non-English POS tagging on Flores-101, CrossSum, and XGLUE datasets across eight languages, without updating model parameters. The approach is cost-effective and scalable for multilingual LLM democratization, reducing reliance on expensive SFT while maintaining strong cross-lingual capabilities; future work includes extending to more models and exploring RLHF-based alignment.

Abstract

While supervised fine-tuning (SFT) has been a straightforward approach for tailoring the output of foundation large language model (LLM) to specific preferences, concerns have been raised about the depth of this alignment, with some critiques suggesting it is merely "superficial". We critically examine this hypothesis within the scope of cross-lingual generation tasks, proposing that the effectiveness of SFT may be constrained by its reliance on prior tokens to guide cross-lingual generation. Based on this crucial insight, and in response to the challenges posed by the costly and limited availability of non-English data for SFT, we introduce a novel training-free alignment method named PreTTY, which employs minimal task-related prior tokens to bridge the foundation LLM and the SFT LLM, achieving comparable performance without training. Experiments on machine translation and part-of-speech tagging across eight languages demonstrate the efficacy of PreTTY in cross-lingual settings. Remarkably, by initiating the decoding process with only one or two prior tokens, foundation LLMs can achieve performance comparable to their SFT counterparts. This method presents a cost-effective alternative to SFT and advances the democratization of multilingual LLMs.

Figures (5)

• Figure 1: Illustration of our research question and proposed Prefix TexT as a Yarn (Pretty) framework.
• Figure 2: The agreement between the SFT model and the foundation model in terms of the selection of the next token. Once the Prior Token is provided, the token chosen by the SFT model is also can be found within the Top-K candidate words of foundation model.
• Figure 3: The probability distribution of tokens selected by various models. Incorporation of a Prior Token causes the decision probabilities of both models to converge across all data instances.
• Figure 4: The divergence in probability distributions across the entire vocabulary during decoding. Prior Token significantly reduces the discrepancy between the foundation model and the SFT model.
• Figure 5: Impact of incrementally adding refined prior tokens on performance across Flores-101 subsets.