CIRCLE: A Framework for Evaluating AI from a Real-World Lens

TL;DR

CIRCLE operationalizes the Validation phase of TEVV by formalizing the translation of stakeholder concerns outside the stack into measurable signals, and provides a structured, prospective protocol for linking context-sensitive qualitative insights to scalable quantitative metrics.

Abstract

This paper proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI's materialized outcomes in deployment. While existing frameworks like MLOps focus on system stability and benchmarks measure abstract capabilities, decision-makers outside the AI stack lack systematic evidence about the behavior of AI technologies under real-world user variability and constraints. CIRCLE operationalizes the Validation phase of TEVV (Test, Evaluation, Verification, and Validation) by formalizing the translation of stakeholder concerns outside the stack into measurable signals. Unlike participatory design, which often remains localized, or algorithmic audits, which are often retrospective, CIRCLE provides a structured, prospective protocol for linking context-sensitive qualitative insights to scalable quantitative metrics. By integrating methods such as field testing, red teaming, and longitudinal studies into a coordinated pipeline, CIRCLE produces systematic knowledge: evidence that is comparable across sites yet sensitive to local context. This can enable governance based on materialized downstream effects rather than theoretical capabilities.

Figures (9)

• Figure 1: The CIRCLE framework, illustrating its iterative structure and the primary outputs produced at each stage.
• Figure 2: Tracing a stakeholder‑specified concern about over‑reliance on an AI chatbot through the full lifecycle, from construct formation to observable behavior and longer‑term outcomes in an EdTech classroom.
• Figure 3: Stage 1 of the lifecycle in the EdTech example: eliciting and systematizing stakeholder concerns about over‑reliance on an AI chatbot, with later stages shown in grey.
• Figure 4: Figure \ref{['fig:stage2-edtech']}. Stage 2 of the lifecycle in the EdTech example: designing test activities to assess the presence of the defined construct.
• Figure 5: Evaluation methods as design choices in Stage 2 of the CIRCLE lifecycle. Methods trade off control and contextual richness, shaping what forms of evidence and downstream effects can be observed. Regions illustrate classes of methods that may be selected and combined during this stage.