Flora: Effortless Context Construction to Arbitrary Length and Scale

TL;DR

Flora tackles the challenge of enabling LLMs to process arbitrarily long contexts without relying on costly human or LLM data generation. It introduces a data-construction framework that concatenates short-context instruction-following data into long-context examples guided by meta-instructions, including four strategies (FQA, ABA, ANA, AID) and seven augmentations, plus a token-length distribution rule $y = 2.411e^{-10.899x}+0.017$ to shape data composition. The authors curate Flora-80k and Flora-128k datasets and fine-tune Llama-3-8B-Instruct and QwQ-32B with LoRA, achieving state-of-the-art results on long-context benchmarks such as LongBench v2 and Needle-In-A-HayStack while largely preserving short-context performance. This approach enables scalable, diverse long-context training with minimal resource overhead and has practical implications for long conversations and document analysis in real-world applications.

Abstract

Effectively handling long contexts is challenging for Large Language Models (LLMs) due to the rarity of long texts, high computational demands, and substantial forgetting of short-context abilities. Recent approaches have attempted to construct long contexts for instruction tuning, but these methods often require LLMs or human interventions, which are both costly and limited in length and diversity. Also, the drop in short-context performances of present long-context LLMs remains significant. In this paper, we introduce Flora, an effortless (human/LLM-free) long-context construction strategy. Flora can markedly enhance the long-context performance of LLMs by arbitrarily assembling short instructions based on categories and instructing LLMs to generate responses based on long-context meta-instructions. This enables Flora to produce contexts of arbitrary length and scale with rich diversity, while only slightly compromising short-context performance. Experiments on Llama3-8B-Instruct and QwQ-32B show that LLMs enhanced by Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks. Our data-construction code is available at \href{https://github.com/txchen-USTC/Flora}{https://github.com/txchen-USTC/Flora}.

Figures (9)

• Figure 1: Average scores across long and short context tasks, normalized by the highest score on each task. The scores on LongBench v2 bai2024longbench are evaluated in the zero-shot + CoT setting. Our Flora-enhanced models achieve state-of-the-art (SOTA) performances on all long and short context tasks, compared to other models of similar parameter scales.
• Figure 2: Comparison of output token length distributions between Flora-enhanced and Mosaic-IT enhanced data under fixed total token lengths. The $x$-axis shows the output token length, and the $y$-axis shows the ratio.
• Figure 3: Illustration of Flora, our effortless long-context construction strategy designed to generate theoretically infinite long-context data without any human or LLM intervention. Flora first categorizes the original short instruction tuning datasets into three domains: math, coding, and general knowledge. By applying targeted long-context data augmentations to each category, Flora can enhance the different long-context capabilities of LLMs.
• Figure 4: We show that the token length distribution of present public long context instruction tuning datasets can be fitted as a curve: $y = 2.411e^{-10.899x}+0.017$, where $x$-axis measures the normalized token length ranges from 0 to 1, and $y$-axis measures the data sample proportion.
• Figure 5: (a) Results of Single-Retrieval task in Needle-In-A-Haystack benchmark. The $x$-axis represents the context lengths, while the $y$-axis indicates the depth of the inserted needle. The green color signifies a score close to 1, and red denotes a score close to 0. (b) Results on five short-context tasks from the Open LLM Leaderboard 2.