SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization

TL;DR

SoLoPO addresses the underutilization of long-context information in LLMs by decoupling long-context preference optimization into short-context preference optimization and short-to-long reward alignment. The authors provide a theoretical upper bound linking long-context loss to short-context loss plus an alignment term, and propose a chosen-only SoLo-RA variant to improve efficiency. Empirically, SoLoPO enhances long-context generalization across multiple benchmarks (LongBenchV1/V2, RULER, NIAH-Plus) while preserving short-context performance, and it yields substantial training efficiency gains, enabling longer trainable contexts. The framework is compatible with DPO, SimPO, and ORPO, and shows consistent improvements over vanilla PO methods, offering a practical pathway to robust long-context reasoning in real-world tasks.

Abstract

Despite advances in pretraining with extended context lengths, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives. To address these limitations, we propose a framework named $\textbf{S}$h$\textbf{o}$rt-to-$\textbf{Lo}$ng $\textbf{P}$reference $\textbf{O}$ptimization ($\textbf{SoLoPO}$), decoupling long-context preference optimization (PO) into two components: short-context PO and short-to-long reward alignment (SoLo-RA), supported by both theoretical and empirical evidence. Specifically, short-context PO leverages preference pairs sampled from short contexts to enhance the model's contextual knowledge utilization ability. Meanwhile, SoLo-RA explicitly encourages reward score consistency utilization for the responses when conditioned on both short and long contexts that contain identical task-relevant information. This facilitates transferring the model's ability to handle short contexts into long-context scenarios. SoLoPO is compatible with mainstream preference optimization algorithms, while substantially improving the efficiency of data construction and training processes. Experimental results show that SoLoPO enhances all these algorithms with respect to stronger length and domain generalization abilities across various long-context benchmarks, while achieving notable improvements in both computational and memory efficiency.

Figures (11)

• Figure 1: Original PO vs. SoLoPO. (a) SoLoPO decouples long-context PO into two components: short-context PO and short-to-long reward alignment, reducing the complexity of preference data construction and minimizing long-text processing during training. (b) Under identical configurations, SoLoPO exhibits superior training efficiency compared to vanilla methods. (c) SoLoPO outperforms the original PO across various long-context benchmarks while maintaining short-context ability.
• Figure 2: Performance improvements of different short-to-long preference optimization frameworks based on various PO algorithms over Qwen2.5-7B on LongBenchV1 (top) and RULER (bottom).
• Figure 3: Performance w/ different $\alpha$ in SoLo-ORPO.
• Figure 4: Run time (RT) and performance gains (Imp.) under varying lengths of $x_{\text{long}}$, with $x_{\text{short}}$ fixed at $1K$.
• Figure 5: Illustration of the construction pipeline for the short-to-long dataset. (1) Irrelevant documents are randomly sampled and concatenated with the original short input to form long contexts. (2) Multiple candidate responses are generated based on the short context and question via the instruct model. (3) Preference pairs are curated using a sub-embased selection guided by ground-truth answers. (4) The final short-to-long dataset, composed of short contexts, long contexts, questions, and preference pairs, is used for training LLM with SoLoPO.

Theorems & Definitions (10)

• Theorem 1: Relation between long-context and short-context preference optimization losses
• Lemma 1
• proof
• proof
• Proposition 1
• proof
• Proposition 2
• proof
• Theorem 2
• proof