An Exploratory Study on How AI Awareness Impacts Human-AI Design Collaboration

TL;DR

This study investigates whether endowing AI with awareness of a designer's activities and current context improves human-AI collaborative design. Using a Wizard-of-Oz setup with within-subject exposure to AI-aware and non-aware conditions, the authors collect quantitative metrics of communication fluency and qualitative interview data from 20 designers. Results show AI awareness increases turn frequency while reducing turn duration and length, with interviews revealing enhanced alignment, reduced effort, and changes in speech style. The authors propose design implications such as adjustable awareness, interruption mechanisms, and speech-centric interaction to guide future human-AI collaborative design systems. Overall, the work highlights the practical value of awareness in making AI a more effective co-designer in dynamic design tasks.

Abstract

The collaborative design process is intrinsically complicated and dynamic, and researchers have long been exploring how to enhance efficiency in this process. As Artificial Intelligence technology evolves, it has been widely used as a design tool and exhibited the potential as a design collaborator. Nevertheless, problems concerning how designers should communicate with AI in collaborative design remain unsolved. To address this research gap, we referred to how designers communicate fluently in human-human design collaboration, and found awareness to be an important ability for facilitating communication by understanding their collaborators and current situation. However, previous research mainly studied and supported human awareness, the possible impact AI awareness would bring to the human-AI collaborative design process, and the way to realize AI awareness remain unknown. In this study, we explored how AI awareness will impact human-AI collaboration through a Wizard-of-Oz experiment. Both quantitative and qualitative results supported that enabling AI to have awareness can enhance the communication fluidity between human and AI, thus enhancing collaboration efficiency. We further discussed the results and concluded design implications for future human-AI collaborative design systems.

Figures (4)

• Figure 1: The interface example of the human-AI design collaborative system: a) Canvas. Canvas is the main interaction area where designers edit information using tools in the toolbar on the bottom right corner, and AI can also upload generated images to the canvas. b) Chat box. Communication history and speech recognition results are displayed in this area, and designers can also type here to initiate conversation in case of need. c) Camera screen. This is presented in the Aware condition only to indicate what AI can see. Note: The interface is a translated version of the original one used for user studies.
• Figure 2: The overview of system structure and data stream. In our study, the participants a) interacted and communicated with the frontend (i.e., the user interface in Figure \ref{['fig:interface']}) and received feedback from AI. The backend consisted of two Wizards and generative models. In the Aware condition, Wizard A would b) obtain awareness information from the camera screen and canvas, c) compose prompts based on predefined scripts and send them to Wizard B. In the Non-aware condition, the prompt sent in process (c) would not contain awareness information. Wizard B in both conditions supported the human-AI communication process by d) processing speech recognition results and input with canvas information, and e) sending prompts coming from the participants and Wizard A to generative models. f) All generation results would be uploaded to the canvas.
• Figure 3: The WoZ experiment setting, including the positions of the participants and experimenters (Wizards), and important devices.
• Figure 4: The overview of the data analysis process. Our raw data mainly came from three sources: The experiment process, the human-AI conversation during the experiments, and interview. These data were processed into valid data for further analysis, including four types of quantitative data corresponding to our predefined metrics, and three types of qualitative data. We analyzed these data and reported how AI awareness impacted communication between human and AI in the Result section (§\ref{['result']}). Furthermore, we discussed the inferences and causalities of quantitative results and concluded design implications for future human-AI collaborative design systems in the Discussion section (§\ref{['Discussion']}).