Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

Alejandro Breen Herrera, Aayush Sheth, Steven G. Xu, Zhucheng Zhan, Charles Wright, Marcus Yearwood, Hongtai Wei, Sudeep Das

TL;DR

A multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations is introduced and two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA are investigated.

Abstract

Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals two underexplored challenges: how to evaluate multi-turn interactions and how to optimize tightly coupled multi-agent systems. Grocery shopping further amplifies these difficulties, as user requests are often underspecified, highly preference-sensitive, and constrained by factors such as budget and inventory. In this paper, we present a practical blueprint for evaluating and optimizing conversational shopping assistants, illustrated through a production-scale AI grocery assistant. We introduce a multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations. Building on this evaluation foundation, we investigate two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA (Shao et al., 2025): (1) Sub-agent GEPA, which optimizes individual agent nodes against localized rubrics, and (2) MAMuT (Multi-Agent Multi-Turn) GEPA (Herrera et al., 2026), a novel system-level approach that jointly optimizes prompts across agents using multi-turn simulation and trajectory-level scoring. We release rubric templates and evaluation design guidance to support practitioners building production CSAs.

Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

TL;DR

A multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations is introduced and two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA are investigated.

Abstract

Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals two underexplored challenges: how to evaluate multi-turn interactions and how to optimize tightly coupled multi-agent systems. Grocery shopping further amplifies these difficulties, as user requests are often underspecified, highly preference-sensitive, and constrained by factors such as budget and inventory. In this paper, we present a practical blueprint for evaluating and optimizing conversational shopping assistants, illustrated through a production-scale AI grocery assistant. We introduce a multi-faceted evaluation rubric that decomposes end-to-end shopping quality into structured dimensions and develop a calibrated LLM-as-judge pipeline aligned with human annotations. Building on this evaluation foundation, we investigate two complementary prompt-optimization strategies based on a SOTA prompt-optimizer called GEPA (Shao et al., 2025): (1) Sub-agent GEPA, which optimizes individual agent nodes against localized rubrics, and (2) MAMuT (Multi-Agent Multi-Turn) GEPA (Herrera et al., 2026), a novel system-level approach that jointly optimizes prompts across agents using multi-turn simulation and trajectory-level scoring. We release rubric templates and evaluation design guidance to support practitioners building production CSAs.
Paper Structure (16 sections, 2 equations, 2 figures, 7 tables, 1 algorithm)

This paper contains 16 sections, 2 equations, 2 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Agent Overview
  3. Rubric Evaluations and Calibrations
  4. Agent Optimization
  5. Sub-agent GEPA
  6. MAMuT GEPA
  7. Joint Optimization of Prompt Bundles.
  8. Simulated User.
  9. MAMuT vs. Sub-agent GEPA.
  10. Continuous Learning, Conclusion, and Future Work
  11. Appendix
  12. Domain Weights
  13. Binary Check Definitions
  14. Judge Prompts: Shopping Execution
  15. Baseline Prompt
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Example trajectory of MAGIC; main agent translates user's request into actionable tasks. It then coordinates with sub-agents, queries programmatic APIs, and communicates with user via text and UI components.
  • Figure 2: Sub-agent GEPA rubric scores vs. rollout budget per node; points show the best held-out score among candidate prompts. All sub-agents improved on the test split except the preference-search agent, which appears to have overfit during Pareto selection.