Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Generative AI in the Lab: Methodological Challenges and Guidelines

Hyerim Park, Khanh Huynh, Malin Eiband, Jeremy Dillmann, Sven Mayer, Michael Sedlmair

TL;DR

This paper tackles the problem of evaluating GenAI in controlled lab settings, where non-deterministic outputs disrupt traditional evaluation assumptions. It adopts a reflective, multi-case approach, analyzing four GenAI-integrated lab studies to identify five recurring methodological challenges (C1–C5) and to develop five guidelines (G1–G5) with eighteen practical recommendations. The contributions include a methodological framework for transparency and comparability in GenAI studies, plus actionable strategies spanning participant preparation, prototype fidelity, feedback interpretability, evaluation metrics, and flexible study design. Collectively, the work provides a foundational, adaptable blueprint for robust GenAI lab evaluations and motivates ongoing methodological reflection as GenAI capabilities evolve.

Abstract

Generative AI (GenAI) systems are inherently non-deterministic, producing varied outputs even for identical inputs. While this variability is central to their appeal, it challenges established HCI evaluation practices that typically assume consistent and predictable system behavior. Designing controlled lab studies under such conditions therefore remains a key methodological challenge. We present a reflective multi-case analysis of four lab-based user studies with GenAI-integrated prototypes, spanning conversational in-car assistant systems and image generation tools for design workflows. Through cross-case reflection and thematic analysis across all study phases, we identify five methodological challenges and propose eighteen practice-oriented recommendations, organized into five guidelines. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by GenAI's stochastic nature: (C1) reliance on familiar interaction patterns, (C2) fidelity-control trade-offs, (C3) feedback and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system issues. Our guidelines address these challenges through strategies such as reframing onboarding to help participants manage unpredictability, extending evaluation with constructs such as trust and intent alignment, and logging system events, including hallucinations and latency, to support transparent analysis. This work contributes (1) a methodological reflection on how GenAI's stochastic nature unsettles lab-based HCI evaluation and (2) eighteen recommendations that help researchers design more transparent, robust, and comparable studies of GenAI systems in controlled settings.

Evaluating Generative AI in the Lab: Methodological Challenges and Guidelines

TL;DR

This paper tackles the problem of evaluating GenAI in controlled lab settings, where non-deterministic outputs disrupt traditional evaluation assumptions. It adopts a reflective, multi-case approach, analyzing four GenAI-integrated lab studies to identify five recurring methodological challenges (C1–C5) and to develop five guidelines (G1–G5) with eighteen practical recommendations. The contributions include a methodological framework for transparency and comparability in GenAI studies, plus actionable strategies spanning participant preparation, prototype fidelity, feedback interpretability, evaluation metrics, and flexible study design. Collectively, the work provides a foundational, adaptable blueprint for robust GenAI lab evaluations and motivates ongoing methodological reflection as GenAI capabilities evolve.

Abstract

Generative AI (GenAI) systems are inherently non-deterministic, producing varied outputs even for identical inputs. While this variability is central to their appeal, it challenges established HCI evaluation practices that typically assume consistent and predictable system behavior. Designing controlled lab studies under such conditions therefore remains a key methodological challenge. We present a reflective multi-case analysis of four lab-based user studies with GenAI-integrated prototypes, spanning conversational in-car assistant systems and image generation tools for design workflows. Through cross-case reflection and thematic analysis across all study phases, we identify five methodological challenges and propose eighteen practice-oriented recommendations, organized into five guidelines. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by GenAI's stochastic nature: (C1) reliance on familiar interaction patterns, (C2) fidelity-control trade-offs, (C3) feedback and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system issues. Our guidelines address these challenges through strategies such as reframing onboarding to help participants manage unpredictability, extending evaluation with constructs such as trust and intent alignment, and logging system events, including hallucinations and latency, to support transparent analysis. This work contributes (1) a methodological reflection on how GenAI's stochastic nature unsettles lab-based HCI evaluation and (2) eighteen recommendations that help researchers design more transparent, robust, and comparable studies of GenAI systems in controlled settings.
Paper Structure (68 sections, 3 figures, 2 tables)

This paper contains 68 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodological Shifts in HCI Evaluation
  4. Evaluating GenAI Systems in HCI
  5. Methodology: A Multi-Case Study with Reflection and Thematic Analysis
  6. Case Selection and Context
  7. Data Collection
  8. Analysis
  9. Step 1: Organizing Case Observations.
  10. Step 2: Collaborative Coding and Clustering.
  11. Step 3: Thematic Synthesis.
  12. Results: Challenges Identified Across Case Studies
  13. C1. GenAI amplifies user reliance on familiar interaction patterns
  14. C2. Trade-offs between fidelity and control are amplified by GenAI's output variability
  15. C3. Redefined feedback loops challenge user trust in GenAI systems
  16. ...and 53 more sections

Figures (3)

  • Figure 1: Visualization of our inductive thematic coding process from initial analytic codes (light gray boxes) to subthemes (white boxes) and five higher-order themes (C1--C5). Each column represents one methodological challenge that emerged through iterative comparison and clustering across studies. Detailed documentation of affinity notes, analytic codes, subthemes, themes, and methodological reflections is provided in the supplementary material.
  • Figure 2: Methodological challenges (C1--C5) and corresponding guidelines (G1--G5) with all eighteen recommendations (R1.1--R5.3) for evaluating GenAI systems in HCI lab studies. The figure illustrates where each recommendation primarily applies across research phases, from planning and prototyping to data collection and analysis.
  • Figure 3: Overview of the initial collaborative workspace used during an early stage of analysis. Methodological observations were externalized and provisionally grouped to support shared sensemaking and explore emerging patterns. These materials were iteratively added, merged, and reorganized to explore potential subthemes and overarching patterns, informed by principles of thematic analysis braun_using_2006. Challenge labels (C1--C5) were added to make the connection between early affinity notes and the resulting challenges explicit.