Exploring Information Seeking Agent Consolidation

TL;DR

This work tackles the consolidation of heterogeneous information-seeking agents—open-web, document-grounded, and knowledge-base retrieval—into a single foundation model. It systematically compares data-level consolidation (joint multi-environment training) with parameter-level consolidation (merging independently trained experts), analyzing robustness, cross-domain generalization, and interference across tasks. Key findings show data-level consolidation as a strong, stable baseline, while parameter-level merging can match or exceed it in some settings, particularly with careful matrix-level merging and data-aware consensus strategies; however, challenges remain for interference, especially in LoRA-based updates. The study provides concrete design principles to guide robust consolidation, including fine-grained merging, normalization of updates, adaptive Task similarity coefficients, data-calibration approaches, and geometry-aware merging, with significant implications for scalable, cross-domain agent systems.

Abstract

Information-seeking agents have emerged as a powerful paradigm for solving knowledge-intensive tasks. Existing information-seeking agents are typically specialized for open web, documents, or local knowledge bases, which constrains scalability and cross-domain generalization. In this work, we investigate how to consolidate heterogeneous information-seeking agents into a single foundation agentic model. We study two complementary consolidation strategies: data-level consolidation, which jointly trains a unified model on a mixture of domain-specific datasets, and parameter-level consolidation, which merges independently trained agent models at the parameter level. Our analysis compares these approaches in terms of performance retention, cross-domain generalization, and interference across information-seeking behaviors. Our results show that data-level consolidation remains a strong and stable baseline, while parameter-level consolidation offers a promising, efficient alternative but suffers from interference and robustness challenges. We further identify key design factors for effective agent consolidation at the parameter level, including fine-grained merging granularity, awareness of task heterogeneity, and principled consensus strategy.

Figures (13)

• Figure 1: Comparison of three information-seeking agent consolidation paradigms. (a) Single-task training, where separate agents are independently trained for local knowledge-base retrieval $\mathcal{D}_{\text{rag}}$, document understanding $\mathcal{D}_{\text{doc}}$, and open-web search $\mathcal{D}_{\text{web}}$, in their respective environments. (b) Data-level consolidation, which unifies heterogeneous agent trajectories into a single training set $\mathcal{D}_{\text{all}}$ and learns a single model via joint multi-task training. (c) Parameter-level consolidation, which first trains environment-specific expert models as (a) and then merges them in parameter space to obtain a unified agent without joint retraining.
• Figure 2: Average tool usage frequency and average answer length across different information-seeking settings in training data.
• Figure 3: Differences of information-seeking behavior between consolidation methods and the expert agent across benchmarks on Qwen3-30B-A3B-think. We select the top-performing parameter-level consolidation method, RegMean++, as the representative. Results are reported across multiple information-seeking categories, with detailed definitions of each category provided in Appendix \ref{['app:behavior_defs']}.
• Figure 4: Layer-wise L2 norm of parameter updates of expert agents for Web, Doc, and RAG agents across model depth.
• Figure 5: Training loss curves and benchmark performance during training of Qwen3-30B-A3B-Think.