Adobe Summit Concierge Evaluation with Human in the Loop

TL;DR

The paper tackles the problem of building a reliable, domain-specific AI assistant for Adobe Summit under data-sparse and rapid-deployment constraints. It adopts a human-in-the-loop workflow combining prompt engineering, retrieval-grounded (RAG) and NL2SQL grounding, and lightweight validation to bootstrap the system with synthetic data and minimal pre-existing logs. Key contributions include a HITL quality assurance loop, data-sparsity bootstrapping techniques, and insights from real-world deployment that demonstrate scalable, reliable performance in cold-start enterprise contexts. The work provides a practical blueprint for developing domain-specific AI assistants that balance accuracy, speed, and user experience in live event environments.

Abstract

Generative AI assistants offer significant potential to enhance productivity, streamline information access, and improve user experience in enterprise contexts. In this work, we present Summit Concierge, a domain-specific AI assistant developed for Adobe Summit. The assistant handles a wide range of event-related queries and operates under real-world constraints such as data sparsity, quality assurance, and rapid deployment. To address these challenges, we adopt a human-in-the-loop development workflow that combines prompt engineering, retrieval grounding, and lightweight human validation. We describe the system architecture, development process, and real-world deployment outcomes. Our experience shows that agile, feedback-driven development enables scalable and reliable AI assistants, even in cold-start scenarios.

Figures (2)

• Figure 1: The overview of Summit concierge.
• Figure 2: Summit Concierge generated response for the query - "What keynotes does Shantanu Narayen speak at?".