PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework
Sina Montazeri, Yunhe Feng, Kewei Sha
TL;DR
PublicAgent introduces a multi-agent framework that decomposes open data analysis into specialized roles—intent clarification, dataset discovery, data analysis, and report generation—under a central orchestrator to mitigate attention dilution and error propagation in end-to-end LLM workflows. Through systematic ablations across five models and 50 queries, the work derives five design principles: specialization provides value independent of model strength, universal versus conditional agent utility, non-redundant failure mitigation, persistent architectural benefits across task complexity, and model-aware architecture design. The evaluation demonstrates that the multi-agent design yields robust performance across diverse domains, with universal agents delivering consistent gains and conditional agents requiring model-aware activation. The practical impact lies in making public data analysis accessible via natural language interfaces while providing actionable guidance for designing multi-agent AI systems for complex, multi-stage workflows.
Abstract
Open data repositories hold potential for evidence-based decision-making, yet are inaccessible to non-experts lacking expertise in dataset discovery, schema mapping, and statistical analysis. Large language models show promise for individual tasks, but end-to-end analytical workflows expose fundamental limitations: attention dilutes across growing contexts, specialized reasoning patterns interfere, and errors propagate undetected. We present PublicAgent, a multi-agent framework that addresses these limitations through decomposition into specialized agents for intent clarification, dataset discovery, analysis, and reporting. This architecture maintains focused attention within agent contexts and enables validation at each stage. Evaluation across five models and 50 queries derives five design principles for multi-agent LLM systems. First, specialization provides value independent of model strength--even the strongest model shows 97.5% agent win rates, with benefits orthogonal to model scale. Second, agents divide into universal (discovery, analysis) and conditional (report, intent) categories. Universal agents show consistent effectiveness (std dev 12.4%) while conditional agents vary by model (std dev 20.5%). Third, agents mitigate distinct failure modes--removing discovery or analysis causes catastrophic failures (243-280 instances), while removing report or intent causes quality degradation. Fourth, architectural benefits persist across task complexity with stable win rates (86-92% analysis, 84-94% discovery), indicating workflow management value rather than reasoning enhancement. Fifth, wide variance in agent effectiveness across models (42-96% for analysis) requires model-aware architecture design. These principles guide when and why specialization is necessary for complex analytical workflows while enabling broader access to public data through natural language interfaces.