Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs

TL;DR

The paper tackles the problem that LLMs often ground outputs poorly in contextual information. It proposes Router Lens to identify context-faithful experts by fine-tuning only the router and computing a Context-dependence Ratio, revealing experts that highly leverage context. Building on this, CEFT fine-tunes only the context-faithful experts, achieving competitive or superior performance with far fewer trainable parameters than full fine-tuning. Extensive experiments across multiple MoE models and context-dependent tasks demonstrate improved context grounding, better robustness against forgetting, and substantial training efficiency. The work advances practical strategies for aspect-specific expert optimization in Mixture-of-Experts LLMs, with implications for reliable reasoning in context-rich settings.

Abstract

Context faithfulness is essential for reliable reasoning in context-dependent scenarios. However, large language models often struggle to ground their outputs in the provided context, resulting in irrelevant responses. Inspired by the emergent expert specialization observed in mixture-of-experts architectures, this work investigates whether certain experts exhibit specialization in context utilization, offering a potential pathway toward targeted optimization for improved context faithfulness. To explore this, we propose Router Lens, a method that accurately identifies context-faithful experts. Our analysis reveals that these experts progressively amplify attention to relevant contextual information, thereby enhancing context grounding. Building on this insight, we introduce Context-faithful Expert Fine-Tuning (CEFT), a lightweight optimization approach that selectively fine-tunes context-faithful experts. Experiments across a wide range of benchmarks and models demonstrate that CEFT matches or surpasses the performance of full fine-tuning while being significantly more efficient.

Figures (14)

• Figure 1: A case from NQ-Swap DBLP:conf/emnlp/LongprePCRD021/nq-swap where MoE experts exhibit different tendencies of context faithfulness (answer is underlined in red).
• Figure 2: Illustration of Router Lens, where the Context-faithful Experts (CE) in each layer are identified by the tuned router network.
• Figure 3: Comparison of the performance impact on NQ-Swap when masking $k$ original experts (OE) vs. the top-$k$ context-faithful experts (CE), evaluated on router-tuned (RT) OLMoE-1B-7B and MiniCPM-MoE-8x2B models, relative to their respective base models (Base).
• Figure 4: t-SNE visualization of context-faithful expert activation patterns in OLMoE-1B-7B and MiniCPM-MoE-8×2B. For each model, 1,000 examples per dataset are randomly selected for projection.
• Figure 5: Cross-task transfer performance of router-tuned models. Each cell shows the absolute EM score improvement over the base model, where the model is trained on the dataset in row $i$ and evaluated on the dataset in column $j$.