UX Remix: Improving Measurement Item Design Process Using Large Language Models and Prior Literature

TL;DR

The paper tackles the lack of a standardized process for designing measurement items in UX evaluation and the challenge of reusing prior items across contexts. It introduces UX Remix, a web-based system that stores 697 literature-derived constructs in a vector database, uses LLMs to generate a custom construct and refined measurement items, and guides researchers through a three-page workflow. The approach facilitates rapid retrieval of relevant constructs, tailored item generation, and contextual refinement, aiming to improve the rigor and efficiency of measurement design in HCI. The work highlights practical benefits and outlines directions for validation, expansion of the item corpus, and improved prompting strategies.

Abstract

Researchers often struggle to develop measurement items and lack a standardized process. To support the design process, we present UX Remix, a system to help researchers develop constructs and measurement items using large language models (LLMs). UX Remix leverages a database of constructs and associated measurement items from previous papers. Based on the data, UX Remix recommends constructs relevant to the research context. The researchers then select appropriate constructs based on the recommendations. Afterward, selected constructs are used to generate a custom construct, and UX Remix recommends measurement items. UX Remix streamlines the process of selecting constructs, developing measurement items, and adapting them to research contexts, addressing challenges in the selection and reuse of measurement items. This paper describes the implementation of the system, the potential benefits, and future directions to improve the rigor and efficiency of measurement design in human-computer interaction (HCI) research.

Figures (4)

• Figure 1: In UX Remix, users (researchers) fill out the project descriptions. (1) The vector database searches for 10 related constructs and recommends them to users based on the project description. (2) Users select the appropriate constructs that would be used as prompts in a large language model (LLM). (3) LLM generates a custom construct and measurement items for evaluation. (4) Users develop measurement items using recommendations of the LLM.
• Figure 2: Project design page of UX Remix. In this page, researchers write a short project description.
• Figure 3: Construct selection page of UX Remix. UX Remix recommends 10 constructs relevant to the project description. Then, researchers select appropriate constructs based on the details.
• Figure 4: Item development page of UX Remix. UX Remix first generates a custom construct based on the selected constructs from previous page. Then, it refines the measurement items from the selected constructs and recommends most appropriate items to the custom construct. After that, researchers could decide most appropriate items based on the recommendation results.