Language Models can Self-Lengthen to Generate Long Texts

TL;DR

Experiments on benchmarks and human evaluations show that Self-Lengthen outperforms existing methods in long-text generation, when applied to top open-source LLMs such as Qwen2 and LLaMA3.

Abstract

Recent advancements in Large Language Models (LLMs) have significantly enhanced their ability to process long contexts, yet a notable gap remains in generating long, aligned outputs. This limitation stems from a training gap where pre-training lacks effective instructions for long-text generation, and post-training data primarily consists of short query-response pairs. Current approaches, such as instruction backtranslation and behavior imitation, face challenges including data quality, copyright issues, and constraints on proprietary model usage. In this paper, we introduce an innovative iterative training framework called Self-Lengthen that leverages only the intrinsic knowledge and skills of LLMs without the need for auxiliary data or proprietary models. The framework consists of two roles: the Generator and the Extender. The Generator produces the initial response, which is then split and expanded by the Extender. This process results in a new, longer response, which is used to train both the Generator and the Extender iteratively. Through this process, the models are progressively trained to handle increasingly longer responses. Experiments on benchmarks and human evaluations show that Self-Lengthen outperforms existing methods in long-text generation, when applied to top open-source LLMs such as Qwen2 and LLaMA3. Our code is publicly available at https://github.com/QwenLM/Self-Lengthen.

Figures (9)

• Figure 1: Existing LLMs are struggling to generate long, aligned outputs. (Left) The actual output lengths when prompting LLMs for generation tasks with specific length constraints under default parameters. All four tested LLMs struggle to exceed 2,000 words. (Right) We attempted to get longer outputs by adjusting the decoding strategies (sampling and beam-search) and parameters (min_tokens, length_penalty, etc.); see \ref{['appx:pilot-study']} for details. However, the quality evaluated by humans significantly deteriorated after reaching 2,000 words. By employing our proposed Self-Lengthen method, we significantly enhanced the output length of the Qwen2-7B-Instruct backbone model while preserving the quality of the generated content.
• Figure 2: A high-level overview of the proposed Self-Lengthen. (Micro-Iteration) We begin by using the Generator to produce an initial response. Then, we employ the Extender to expand this response in a two-stage process, resulting in a much longer output. This extension process can be repeated iteratively to create increasingly lengthy responses. (Macro-Iteration) Once we have generated long responses, we use them to fine-tune both the Generator and Extender. These improved models are then utilized in subsequent iterations to generate and extend even longer responses.
• Figure 3: An illustration of the response extension process. Given an instruction, we first employ the Generator to produce an initial response. We then use the Extender to take the first half of this initial response to create the extension of the first part (Stage 1). Following this, we proceed to extend the entire response, using the previous two-thirds of the extended first half for demonstration to complete the extension of the remaining part (Stage 2). Ultimately, the model delivers a cohesive and consistent extended response. Note that this example is just for illustrative purposes since in real scenarios the responses would be much longer.
• Figure 3: Key statistics of LonGen, which contains 2 languages * 3 length ranges * 4 constraint types * 10 = 240 pieces of data in total.
• Figure 4: The results of data evaluation.