CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems

TL;DR

MACGS produce extensive intermediate outputs whose contributions to final correctness are opaque. CAM introduces a causality-based framework that models these outputs as structured features, builds a causal graph, and uses counterfactual interventions with influence sets to identify feature importance, enabling targeted improvements. Key findings include context-dependent features, up to 7.2% Pass@1 gains from hybrid backends, and practical downstream tools such as 73.3% success in causality-guided failure repair and up to 66.8% token reduction via pruning. The approach generalizes across MACGS like Self-Collab, and offers actionable guidance for design and deployment, improving robustness and efficiency of multi-agent code generation systems.

Abstract

Despite the remarkable success that Multi-Agent Code Generation Systems (MACGS) have achieved, the inherent complexity of multi-agent architectures produces substantial volumes of intermediate outputs. To date, the individual importance of these intermediate outputs to the system correctness remains opaque, which impedes targeted optimization of MACGS designs. To address this challenge, we propose CAM, the first \textbf{C}ausality-based \textbf{A}nalysis framework for \textbf{M}ACGS that systematically quantifies the contribution of different intermediate features for system correctness. By comprehensively categorizing intermediate outputs and systematically simulating realistic errors on intermediate features, we identify the important features for system correctness and aggregate their importance rankings. We conduct extensive empirical analysis on the identified importance rankings. Our analysis reveals intriguing findings: first, we uncover context-dependent features\textemdash features whose importance emerges mainly through interactions with other features, revealing that quality assurance for MACGS should incorporate cross-feature consistency checks; second, we reveal that hybrid backend MACGS with different backend LLMs assigned according to their relative strength achieves up to 7.2\% Pass@1 improvement, underscoring hybrid architectures as a promising direction for future MACGS design. We further demonstrate CAM's practical utility through two applications: (1) failure repair which achieves a 73.3\% success rate by optimizing top-3 importance-ranked features and (2) feature pruning that reduces up to 66.8\% intermediate token consumption while maintaining generation performance. Our work provides actionable insights for MACGS design and deployment, establishing causality analysis as a powerful approach for understanding and improving MACGS.

Figures (5)

• Figure 1: Illustration of causality.
• Figure 2: Feature responsibility (FR) distribution across different LLMs on CoderEval. (1) -- (3) represent top-3 FR-ranked features.
• Figure 3: Feature responsibility (FR) distribution across different datasets on Deepseek.
• Figure 4: Results of Self-Collab.
• Figure : Computing Important Feature Set for One Coding Problem