Evaluation and Incident Prevention in an Enterprise AI Assistant

TL;DR

Enterprise AI assistants face high-stakes errors that can constitute incidents. The paper proposes a three-pronged framework: a severity-based incident taxonomy, scalable benchmarking with shared evaluation datasets, and a continual-improvement cycle using multi-source signals. Key contributions include a Sev-0/1/2 taxonomy with a decision-tree for robust annotation, covariate-aware coresets and the GIGA algorithm to reduce labeling effort, and adversarial testing plus quarterly holdout datasets to forecast end-to-end impact. The approach enables proactive incident prevention, cross-team alignment, and systematic improvements in accuracy, verifiability, and user experience, paving the way for more trustworthy enterprise AI systems.

Abstract

Enterprise AI Assistants are increasingly deployed in domains where accuracy is paramount, making each erroneous output a potentially significant incident. This paper presents a comprehensive framework for monitoring, benchmarking, and continuously improving such complex, multi-component systems under active development by multiple teams. Our approach encompasses three key elements: (1) a hierarchical ``severity'' framework for incident detection that identifies and categorizes errors while attributing component-specific error rates, facilitating targeted improvements; (2) a scalable and principled methodology for benchmark construction, evaluation, and deployment, designed to accommodate multiple development teams, mitigate overfitting risks, and assess the downstream impact of system modifications; and (3) a continual improvement strategy leveraging multidimensional evaluation, enabling the identification and implementation of diverse enhancement opportunities. By adopting this holistic framework, organizations can systematically enhance the reliability and performance of their AI Assistants, ensuring their efficacy in critical enterprise environments. We conclude by discussing how this multifaceted evaluation approach opens avenues for various classes of enhancements, paving the way for more robust and trustworthy AI systems.

Figures (5)

• Figure 1: A concrete implementation of how error severity is derived via a series of (more) objective human annotations
• Figure 2: An example of the number of annotations and interactions occurring after the public release of a compound AI Assistant. The specific $x$ and $y$ axis values have been omitted due to privacy concerns.
• Figure 3: Mean squared error of uniform (random) sampling and covariate aware sampling using GIGA. Error is measured with respect to proportion estimates obtained over the full set of annotations.
• Figure 4: Creation of shared evaluation datasets on an ongoing basis, using sampling and human annotation of production traffic, which is then partitioned into development and holdout datasets.
• Figure 5: The continual improvement framework, emphasizing human annotation as a way of both generating labeled data to be used in shared evaluation datasets, and in driving measurement and error analysis. With the Error-severity framework, we are then able to prioritize improved AI components, but also consider other improvements like UX changes that aid in verifiability, explainability, and enhancing user's ability to recover.