Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data

TL;DR

The paper addresses how networks of interacting LLMs perform CollaborativeQA and how hallucinations propagate through network structure. It introduces a coupled mean-field dynamics (MFD) and randomized utility model (RUM) to model information diffusion and LLM decision-making, yielding an analytically tractable ODE framework and a data-driven transition kernel. Theoretical results establish fixed-point existence, contraction, and monotone comparative statics in the incentive, while extensive experiments with 100 open-source LLMs demonstrate that computation, data placement, topology (notably power-law networks), and higher-capability nodes significantly improve the truthful fraction ρ_T. The work provides a principled lens for designing robust, scalable LLM networks for CQA with implications for privacy, fault tolerance, and distributed reasoning in real-world settings.

Abstract

In this paper, we model and analyze how a network of interacting LLMs performs collaborative question-answering (CQA) in order to estimate a ground truth given a distributed set of documents. This problem is interesting because LLMs often hallucinate when direct evidence to answer a question is lacking, and these effects become more pronounced in a network of interacting LLMs. The hallucination spreads, causing previously accurate LLMs to hallucinate. We study interacting LLMs and their hallucination by combining novel ideas of mean-field dynamics (MFD) from network science and the randomized utility model from economics to construct a useful generative model. We model the LLM with a latent state that indicates if it is truthful or not with respect to the ground truth, and extend a tractable analytical model considering an MFD to model the diffusion of information in a directed network of LLMs. To specify the probabilities that govern the dynamics of the MFD, we propose a randomized utility model. For a network of LLMs, where each LLM has two possible latent states, we posit sufficient conditions for the existence and uniqueness of a fixed point and analyze the behavior of the fixed point in terms of the incentive (e.g., test-time compute) given to individual LLMs. We experimentally study and analyze the behavior of a network of $100$ open-source LLMs with respect to data heterogeneity, node capability, network structure, and sensitivity to framing on multiple semi-synthetic datasets.

Figures (8)

• Figure 1: This paper proposes an analytical model for a network of interacting LLMs performing state estimation - we model the information diffusion in the network using a mean-field dynamics (MFD) for directed networks and model the utility of an LLM as being sampled from a random utility model (RUM), which allows for a transition model which can be plugged into the MFD. We further empirically analyze the behavior of a network of 100 open-source interacting LLMs for CQA on different semi-synthetic datasets.
• Figure 2: Illustrating the predictive capabilities of the Mean-Field ODE for a network comprising $100$ LLMs specified in \ref{['eq:dynamic equation for evolution of latent state distribution']}. The mean-field ODE with the RUM model accurately predicts the dynamics of the population state for different questions by estimating the parameters from the first $150$ interactions, enabling application of systematic analysis, simulation-based studies, and control theoretic frameworks.
• Figure 3: Evolution of population state $\boldsymbol{\rho}$ (vs interaction rounds) for the three CQA datasets with different numbers of deliberation rounds and different levels of heterogeneity in the network. As intuitively expected, increasing the number of deliberation rounds results in a better outcome (fraction of truthful LLMs).
• Figure 4: For different networks (chain, power-law and tree), the placement of context affects the fixed point of the population state that the LLM network converges to: if the correct context is placed on more influential nodes the network converges to a higher $\boldsymbol{\rho}_T$ (truthful proportion of LLMs). When the data is randomly assigned to LLMs in a network, then we observe that a network with a power law distribution has higher variability compared to chain and tree network structures.
• Figure 5: For the three datasets, (event, fiction and cutoff), the strength of the influential node affects the convergence of the population state $\boldsymbol{\rho}$: More LLMs converge to the truth if a stronger LLM (8 billion versus 3 billion parameters) has a higher degree centrality.

Theorems & Definitions (2)

• Theorem 1: Fixed point and comparative statics under state-dependent RUM
• proof