Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning

Yuheng Lu, ZiMeng Bai, Caixia Yuan, Huixing Jiang, Xiaojie Wang

TL;DR

This paper addresses the challenge of complex instruction following in large language models, where standard SFT often neglects sub-contexts within multi-constraint instructions. It introduces MISO, a Multi-Input-Single-Output extension for decoder-only transformers, which processes multiple input sequences (sub-contexts) and fuses their representations through a weighted MISO_CausalAttention so that output tokens still align with full instructions. Empirical results on multi-constraint benchmarks and general complex-instruction datasets show that MISO-para (parallel inputs) often outperforms standard SFT, with MISO-succ (chunked inputs) offering additional efficiency gains and strong performance in joint-sub-context tasks. The approach remains compatible with existing LLM inference pipelines and suggests potential extensions to RAG-like settings, indicating practical impact for building more reliable and scalable instruction-following systems.$

Abstract

Large language models (LLMs) exhibit remarkable capabilities in handling natural language tasks; however, they may struggle to consistently follow complex instructions including those involve multiple constraints. Post-training LLMs using supervised fine-tuning (SFT) is a standard approach to improve their ability to follow instructions. In addressing complex instruction following, existing efforts primarily focus on data-driven methods that synthesize complex instruction-output pairs for SFT. However, insufficient attention allocated to crucial sub-contexts may reduce the effectiveness of SFT. In this work, we propose transforming sequentially structured input instruction into multiple parallel instructions containing subcontexts. To support processing this multi-input, we propose MISO (Multi-Input Single-Output), an extension to currently dominant decoder-only transformer-based LLMs. MISO introduces a mixture-of-contexts paradigm that jointly considers the overall instruction-output alignment and the influence of individual sub-contexts to enhance SFT effectiveness. We apply MISO fine-tuning to complex instructionfollowing datasets and evaluate it with standard LLM inference. Empirical results demonstrate the superiority of MISO as a fine-tuning method for LLMs, both in terms of effectiveness in complex instruction-following scenarios and its potential for training efficiency.

Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuning

TL;DR

This paper addresses the challenge of complex instruction following in large language models, where standard SFT often neglects sub-contexts within multi-constraint instructions. It introduces MISO, a Multi-Input-Single-Output extension for decoder-only transformers, which processes multiple input sequences (sub-contexts) and fuses their representations through a weighted MISO_CausalAttention so that output tokens still align with full instructions. Empirical results on multi-constraint benchmarks and general complex-instruction datasets show that MISO-para (parallel inputs) often outperforms standard SFT, with MISO-succ (chunked inputs) offering additional efficiency gains and strong performance in joint-sub-context tasks. The approach remains compatible with existing LLM inference pipelines and suggests potential extensions to RAG-like settings, indicating practical impact for building more reliable and scalable instruction-following systems.$

Abstract

Large language models (LLMs) exhibit remarkable capabilities in handling natural language tasks; however, they may struggle to consistently follow complex instructions including those involve multiple constraints. Post-training LLMs using supervised fine-tuning (SFT) is a standard approach to improve their ability to follow instructions. In addressing complex instruction following, existing efforts primarily focus on data-driven methods that synthesize complex instruction-output pairs for SFT. However, insufficient attention allocated to crucial sub-contexts may reduce the effectiveness of SFT. In this work, we propose transforming sequentially structured input instruction into multiple parallel instructions containing subcontexts. To support processing this multi-input, we propose MISO (Multi-Input Single-Output), an extension to currently dominant decoder-only transformer-based LLMs. MISO introduces a mixture-of-contexts paradigm that jointly considers the overall instruction-output alignment and the influence of individual sub-contexts to enhance SFT effectiveness. We apply MISO fine-tuning to complex instructionfollowing datasets and evaluate it with standard LLM inference. Empirical results demonstrate the superiority of MISO as a fine-tuning method for LLMs, both in terms of effectiveness in complex instruction-following scenarios and its potential for training efficiency.
Paper Structure (42 sections, 6 equations, 2 figures, 4 tables)

This paper contains 42 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. SFT for Instruction Following
  4. Multi-Input-Single-Output Transformers
  5. Prelimilaries
  6. Multi-Constraint Instruction Following
  7. QKV Attention in Transformers
  8. Method
  9. MISO Modeling Structure
  10. Position Ids
  11. MISO Causal Attention
  12. MISO for Complex Instruction Following
  13. Inputs and Position Ids
  14. MISO Causal Attention
  15. Design Behind MISO Causal-Attention
  16. ...and 27 more sections

Figures (2)

  • Figure 1: a) Example multi-constraint instruction following data; b) Attention allocation and whether instruction following success for multi-constraint input and corresponding single-constraint datas; c) Modeling structure for stardard LLM and MISO.
  • Figure 2: Overall structure of MISO as a extension of typical transformer decoder block. Position ids for both parallel input (MISO-para) and successive input (MISO-succ) settings are illustrated.