Efficient Leave-one-out Approximation in LLM Multi-agent Debate Based on Introspection
Yue Cui, Liuyi Yao, Zitao Li, Yaliang Li, Bolin Ding, Xiaofang Zhou
TL;DR
This paper addresses the high computational cost of leave-one-out (LOO) evaluation in LLM-based multi-agent debates by introducing IntrospecLOO, a prompting-based introspective round added after standard debates. IntrospecLOO prompts each agent to disregard a designated peer and update its answer, eliminating the need to re-run an entire debate and reducing token complexity from $O(RTN^2)$ to $O(RN)$. Empirical validation on GSM, MMLU, and Biography datasets shows that IntrospecLOO closely tracks LOO trends across different group sizes and agent compositions, with robust performance across GPT-3.5-turbo and Baichuan models. The method offers a cost-effective, scalable alternative for contribution evaluation, enabling more reliable system refinement and decision-making in large-scale, multi-agent LLM environments.
Abstract
Multi-agent systems based on large language models (LLMs) advance automatic task completion in various fields, where debate is a common cooperation form for agents to solve complicated problems with reasoning and cross-review to solidify answers. Assessing the individual contributions of agents within these debates is crucial for system refinement and outcome reliability. Traditional leave-one-out (LOO) method offers a clear framework for evaluating each agent's role but face challenges in LLM-based systems due to high computational costs and associated financial implications. This paper presents introspective-leave-one-out (IntrospecLOO), a simple yet effective prompting for approximation of LOO in LLM-powered multi-agent debates. IntrospecLOO introduces an additional querying round after standard debates, prompting agents to update their answers while ignoring responses from a designated agent. This strategy effectively isolates and gauges each participant's influence at a reduced query complexity compared to the original LOO approaches. Validation through experiments on three benchmark datasets confirms the effectiveness of IntrospecLOO.