Insight Agents: An LLM-Based Multi-Agent System for Data Insights

TL;DR

The paper addresses the challenge of helping e-commerce sellers discover and leverage data tools by introducing Insight Agents (IA), a hierarchical LLM-driven multi-agent system that delivers personalized data insights through automated retrieval. IA follows a plan-and-execute workflow managed by a dedicated Manager Agent for Out-of-Domain detection and routing, with two worker branches (Data Presenter and Insight Generator) grounded in a Retrieval-Augmented Generation (RAG) framework over tabular data APIs, and enhanced by dynamic domain-knowledge injection. An Auto-encoder-based OOD detector uses the threshold $\mu_{id} + \lambda \sigma_{id}$ to gate queries, while a lightweight $≈33$M-parameter BERT model routes tasks and a query augmenter reduces ambiguity. End-to-end experiments on Amazon US data show high effectiveness, achieving about 90% accuracy and a P90 latency below 15 seconds, with detailed results indicating superior precision for OOD detection and routing, and strong human-evaluated quality (reliability, relevance, and completeness), highlighting IA’s potential as a scalable decision-support platform for e-commerce.

Abstract

Today, E-commerce sellers face several key challenges, including difficulties in discovering and effectively utilizing available programs and tools, and struggling to understand and utilize rich data from various tools. We therefore aim to develop Insight Agents (IA), a conversational multi-agent Data Insight system, to provide E-commerce sellers with personalized data and business insights through automated information retrieval. Our hypothesis is that IA will serve as a force multiplier for sellers, thereby driving incremental seller adoption by reducing the effort required and increase speed at which sellers make good business decisions. In this paper, we introduce this novel LLM-backed end-to-end agentic system built on a plan-and-execute paradigm and designed for comprehensive coverage, high accuracy, and low latency. It features a hierarchical multi-agent structure, consisting of manager agent and two worker agents: data presentation and insight generation, for efficient information retrieval and problem-solving. We design a simple yet effective ML solution for manager agent that combines Out-of-Domain (OOD) detection using a lightweight encoder-decoder model and agent routing through a BERT-based classifier, optimizing both accuracy and latency. Within the two worker agents, a strategic planning is designed for API-based data model that breaks down queries into granular components to generate more accurate responses, and domain knowledge is dynamically injected to to enhance the insight generator. IA has been launched for Amazon sellers in US, which has achieved high accuracy of 90% based on human evaluation, with latency of P90 below 15s.

Figures (3)

• Figure 1: Overall IA Architecture. It illustrates the overall hierarchical structure, consisting of a manager agent overseeing two subordinate worker agents: data presenter agent and insight generator agent.
• Figure 2: Architecture design for data presenter and insight generator.
• Figure 3: Illustration for data workflow planner. Use data presenter as an example.