Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness

Phat Nguyen, Thang Pham

Abstract

Multi-agent systems based on large language models (LLMs) for financial trading have grown rapidly since 2023, yet the field lacks a shared framework for understanding what drives performance or for evaluating claims credibly. This survey makes three contributions. First, we introduce a four-dimensional taxonomy, covering architecture pattern, coordination mechanism, memory architecture, and tool integration; applied to 12 multi-agent systems and two single-agent baselines. Second, we formulate the Coordination Primacy Hypothesis (CPH): inter-agent coordination protocol design is a primary driver of trading decision quality, often exerting greater influence than model scaling. CPH is presented as a falsifiable research hypothesis supported by tiered structural evidence rather than as an empirically validated conclusion; its definitive validation requires evaluation infrastructure that does not yet exist in the field. Third, we document five pervasive evaluation failures (look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, and regime-shift blindness) and show that these can reverse the sign of reported returns. Building on the CPH and the evaluation critique, we introduce the Coordination Breakeven Spread (CBS), a metric for determining whether multi-agent coordination adds genuine value net of transaction costs, and propose minimum evaluation standards as prerequisites for validating the CPH.

Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness

Abstract

Multi-agent systems based on large language models (LLMs) for financial trading have grown rapidly since 2023, yet the field lacks a shared framework for understanding what drives performance or for evaluating claims credibly. This survey makes three contributions. First, we introduce a four-dimensional taxonomy, covering architecture pattern, coordination mechanism, memory architecture, and tool integration; applied to 12 multi-agent systems and two single-agent baselines. Second, we formulate the Coordination Primacy Hypothesis (CPH): inter-agent coordination protocol design is a primary driver of trading decision quality, often exerting greater influence than model scaling. CPH is presented as a falsifiable research hypothesis supported by tiered structural evidence rather than as an empirically validated conclusion; its definitive validation requires evaluation infrastructure that does not yet exist in the field. Third, we document five pervasive evaluation failures (look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, and regime-shift blindness) and show that these can reverse the sign of reported returns. Building on the CPH and the evaluation critique, we introduce the Coordination Breakeven Spread (CBS), a metric for determining whether multi-agent coordination adds genuine value net of transaction costs, and propose minimum evaluation standards as prerequisites for validating the CPH.

Paper Structure

This paper contains 32 sections, 1 equation, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. A Taxonomy of Design Patterns
  4. System Selection
  5. Four Dimensions
  6. D1: Architecture Pattern.
  7. D2: Coordination Mechanism.
  8. D3: Memory Architecture.
  9. D4: Tool Integration.
  10. Cross-System Observations
  11. Evaluation Failures in the Published Literature
  12. Look-Ahead Bias
  13. Survivorship Bias
  14. Backtesting Overfitting
  15. Transaction Cost Neglect
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Taxonomy of LLM-based multi-agent financial systems across four design dimensions. FinVision, ElliottAgents, REITs System, and Agentic RAG are not listed in Coordination Mechanism because they simply utilize a sequential pass-through (minimal coordination).
  • Figure 2: Conceptual illustration of the CBS threshold. Instruments in the upper-left region (low spread, high coordination gain) are candidates for multi-agent coordination; those in the lower-right are not. Asset-class regions are directional and illustrative only; empirical CBS values cannot be computed from currently published results owing to transaction cost neglect (Section \ref{['subsec:txcost']}).