Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Productivity: Rethinking the Impact of Creativity Support Tools

Samuel Rhys Cox, Helena Bøjer Djernæs, Niels van Berkel

TL;DR

This paper investigates how outcomes of Creativity Support Tools (CSTs) are measured in empirical studies, arguing that current CST evaluations overemphasize user experience and artefact quality while neglecting user-centric benefits. The authors surveyed 173 CST evaluations from the ACM Digital Library spanning 2015–2024 and performed a thematic analysis to categorize outcome measures into three themes: User Experience with CST, Creative Artefact Quality, and User-Centric Benefits. They quantify the prevalence of measures, identify widely used instruments such as CSI and TTCT-based metrics, and highlight the growing use of human-AI collaboration, while noting a paucity of validated, CST-specific measures for generative AI and longitudinal assessment. The paper concludes with a call for holistic evaluation practices that incorporate well-being, reflection, and learning, enabling more user-centered CST design and robust, replicable validation frameworks.

Abstract

Creativity Support Tools (CSTs) are widely used across diverse creative domains, with generative AI recently increasing the abilities of CSTs. To better understand how the success of CSTs is determined in the literature, we conducted a review of outcome measures used in CST evaluations. Drawing from (n=173) CST evaluations in the ACM Digital Library, we identified the metrics commonly employed to assess user interactions with CSTs. Our findings reveal prevailing trends in current evaluation practices, while exposing underexplored measures that could broaden the scope of future research. Based on these results, we argue for a more holistic approach to evaluating CSTs, encouraging the HCI community to consider not only user experience and the quality of the generated output, but also user-centric aspects such as self-reflection and well-being as critical dimensions of assessment. We also highlight a need for validated measures specifically suited to the evaluation of generative AI in CSTs.

Beyond Productivity: Rethinking the Impact of Creativity Support Tools

TL;DR

This paper investigates how outcomes of Creativity Support Tools (CSTs) are measured in empirical studies, arguing that current CST evaluations overemphasize user experience and artefact quality while neglecting user-centric benefits. The authors surveyed 173 CST evaluations from the ACM Digital Library spanning 2015–2024 and performed a thematic analysis to categorize outcome measures into three themes: User Experience with CST, Creative Artefact Quality, and User-Centric Benefits. They quantify the prevalence of measures, identify widely used instruments such as CSI and TTCT-based metrics, and highlight the growing use of human-AI collaboration, while noting a paucity of validated, CST-specific measures for generative AI and longitudinal assessment. The paper concludes with a call for holistic evaluation practices that incorporate well-being, reflection, and learning, enabling more user-centered CST design and robust, replicable validation frameworks.

Abstract

Creativity Support Tools (CSTs) are widely used across diverse creative domains, with generative AI recently increasing the abilities of CSTs. To better understand how the success of CSTs is determined in the literature, we conducted a review of outcome measures used in CST evaluations. Drawing from (n=173) CST evaluations in the ACM Digital Library, we identified the metrics commonly employed to assess user interactions with CSTs. Our findings reveal prevailing trends in current evaluation practices, while exposing underexplored measures that could broaden the scope of future research. Based on these results, we argue for a more holistic approach to evaluating CSTs, encouraging the HCI community to consider not only user experience and the quality of the generated output, but also user-centric aspects such as self-reflection and well-being as critical dimensions of assessment. We also highlight a need for validated measures specifically suited to the evaluation of generative AI in CSTs.
Paper Structure (27 sections, 1 figure, 3 tables)

This paper contains 27 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Definition
  5. Inclusion criteria
  6. Exclusion criteria
  7. Search
  8. Selection and Analysis
  9. Findings
  10. User Experience with CST
  11. General Usability Measures:
  12. Measures of Creativity Support:
  13. Interaction Time and Patterns:
  14. Collaboration with Human or AI:
  15. Workload and Cognitive Load:
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Count of measures used (User Experience with CST/Creative Artefact Quality/User-Centric Benefits) by year (2015–2024). As some studies use measures across multiple themes, sums will exceed total number of papers.