How Do We Evaluate Experiences in Immersive Environments?

TL;DR

The paper addresses fragmentation in immersive experience evaluation by conducting a bottom-up scoping review of 375 empirical papers across seven premier HCI/XR venues from 1995 to 2024. It maps constructs, measures, and design goals, showing that practices are domain-sensitive and remain largely researcher-centered, with limited open infrastructure for comparability. The authors identify five evaluation approaches (questionnaires, behavioral measures, system metrics, physiological signals, and interviews) and analyze how instrument choice aligns with device, contribution type, and domain, arguing for smarter, integrated, user-centered methods and open, FAIR-enabled evaluation ecosystems. They propose a forward-looking agenda: promote open protocols and shared datasets, combine computational modeling with user-centered data, and foster an ecosystem that supports reproducibility and cross-study comparability. The work is practically significant for researchers and practitioners aiming to design more coherent, transparent, and sustainable evaluation practices in immersive technologies.

Abstract

How do we evaluate experiences in immersive environments? Despite decades of research in immersive technologies such as virtual reality, the field remains fragmented. Studies rely on overlapping constructs, heterogeneous instruments, and little agreement on what counts as immersive experience. To better understand this landscape, we conducted a bottom-up scoping review of 375 papers published in ACM CHI, UIST, VRST, SUI, IEEE VR, ISMAR, and TVCG. Our analysis reveals that evaluation practices are often domain- and purpose-specific, shaped more by local choices than by shared standards. Yet this diversity also points to new directions. Instead of multiplying instruments, researchers benefit from integrating and refining them into smarter measures. Rather than focusing only on system outputs, evaluations must center the user's lived experience. Computational modeling offers opportunities to bridge signals across methods, but lasting progress requires open and sustainable evaluation practices that support comparability and reuse. Ultimately, our contribution is to map current practices and outline a forward-looking agenda for immersive experience research.

Figures (7)

• Figure 1: PRISMA flowchart detailing the review process for identifying, screening, and including studies in the review. A total of 721 records were identified from database searches. After screening titles and abstracts, 315 records were excluded based on exclusion criteria (EC1–EC4). Of the 406 records sought for full-text retrieval, all were retrieved successfully, with 31 further excluded following full-paper assessment. Ultimately, 375 articles were included in the final review.
• Figure 2: Overview of the reviewed literature on immersive interaction technologies, categorized by publication venues, device types, primary contributions, and application domains. The diagram illustrates the flow of research contributions from key venues (IEEE TVCG, IEEE VR, IEEE ISMAR, ACM CHI, ACM UIST, ACM VRST and ACM SUI
• Figure 3: Categorization of studies based on four classification methods. The total number of studies is consistent across all charts, which categorize the data according to the following criteria: (a) Conference, (b) Equipment, (c) Primary Contribution, and (d) Application Domain.
• Figure 4: Annual distribution of publications from 1995 to 2024. Our results illustrate a significant upward trend in research interest over the last decade, with an exponential surge in publications observed in 2023 and 2024.
• Figure 5: Proportion of papers by number of UX evaluation methods used. The majority of studies adopted either two methods ($N = 158$, 42.1%) or three methods ($N = 121$, 32.3%), suggesting a growing preference for mixed-method evaluation strategies. A smaller portion of papers relied on only one method ($N = 70$, 18.7%), while only 4.5% used four methods ($N = 17$). Notably, 2.4% of papers ($N = 9$) did not report any user experience evaluation. No paper in our dataset used all evaluation methods.