Agent-centric Information Access

TL;DR

This work addresses the problem of information access in a future with millions of domain-specific LLMs by proposing agent-centric information access, where domain experts (knowledge agents) and personalized user agents are dynamically orchestrated via a belief model over expertise $K_1, \dots, K_M$ and the expert pool $L$. It surveys the challenges of expert selection, cross-model answer aggregation, robustness to bias and adversarial manipulation, and evaluation, proposing a scalable framework that leverages retrieval-augmented generation and clustering to simulate thousands of specialized LLMs. A formal evaluation framework is introduced for ranking LLMs, including a reusable training/test collection and an approach to simulate thousands of expert LLMs using clustered document collections and RAG, with new metrics and test designs tailored to model-centric retrieval. The work highlights the practical significance of scalable, cost-aware querying, transparent attribution, and robust aggregation in enabling reliable, scalable multi-LLM information access suitable for deployment at web-scale.

Abstract

As large language models (LLMs) become more specialized, we envision a future where millions of expert LLMs exist, each trained on proprietary data and excelling in specific domains. In such a system, answering a query requires selecting a small subset of relevant models, querying them efficiently, and synthesizing their responses. This paper introduces a framework for agent-centric information access, where LLMs function as knowledge agents that are dynamically ranked and queried based on their demonstrated expertise. Unlike traditional document retrieval, this approach requires inferring expertise on the fly, rather than relying on static metadata or predefined model descriptions. This shift introduces several challenges, including efficient expert selection, cost-effective querying, response aggregation across multiple models, and robustness against adversarial manipulation. To address these issues, we propose a scalable evaluation framework that leverages retrieval-augmented generation and clustering techniques to construct and assess thousands of specialized models, with the potential to scale toward millions.

Figures (3)

• Figure 1: An illustration of a framework for querying both domain-specialized LLMs (knowledge agents) and user-specific LLMs (user agents). Users interact with the system through personalized user agents, which not only track past queries and responses to refine retrieval but also share specialized user knowledge with others. A belief model on expertise determines which knowledge agents ($K_1, K_2, \dots, K_M$) and user agents are experts on specific topics and should be queried to generate the most relevant answers. The system optimizes for cost and latency, ensuring efficient and accurate responses while minimizing unnecessary queries. This architecture enables the dynamic orchestration of expert LLMs, moving beyond document retrieval toward a multi-expert synthesis of knowledge.
• Figure 2: Using RAG-based LLMs to obtain expert models. Each expert consists of a shared LLM and retriever but is restricted to a distinct document collection, specifically structured KBs, which define its domain expertise.
• Figure 3: Word clouds of 4 randomly chosen clusters, each simulating an LLM expert.