Using CognitIDE to Capture Developers' Cognitive Load via Physiological Activity During Everyday Software Development Tasks

TL;DR

Measuring developers' cognitive load in real IDE settings is difficult with purely subjective or code-metric approaches. The authors propose CognitIDE, an IntelliJ-based plugin that streams physiological data via Lab Streaming Layer and maps signals to specific code elements and UI, enabling automated, in-IDE cognitive-load studies. In a feasibility study with four participants, cognitive-load indicators such as pupil dilation, skin conductance responses, and EEG alpha/theta changes increased during tasks vs. baseline, and gaze-driven highlighting aligned with where participants looked. The results suggest that CognitIDE can support ecologically valid investigations of developer experience and code quality, with potential to optimize productivity and well-being in real-world workflows.

Abstract

Integrated development environments (IDE) support developers in a variety of tasks. Unobtrusively capturing developers' cognitive load while working on different programming tasks could help optimize developers' work experience, increase their productivity, and positively impact code quality. In this paper, we propose a study in which the IntelliJ-based IDE plugin CognitIDE is used to collect, map, and visualize software developers' physiological activity data while they are working on various software development tasks. In a feasibility study, participants completed four simulated everyday working tasks of software developers - coding, debugging, code documentation, and email writing - based on Java open source code in the IDE whilst their physiological activity was recorded. Between the tasks, the participants' perceived workload was assessed. Feasibility testing showed that CognitIDE could successfully be used for data collection sessions of one hour, which was the most extended duration tested and was well-perceived by those working with it. Furthermore, the recorded physiological activity indicated higher cognitive load during working tasks compared to baseline recordings. This suggests that cognitive load can be assessed, mapped to code positions, visualized, and discussed with participants in such study setups with CognitIDE. These promising results indicate the usefulness of the plugin for diverse study workflows in a natural IDE environment.

Figures (2)

• Figure 1: Overview of the study procedure. As a first step, the participants were informed about the study and signed a consent form. After all wearables were attached and calibrated, CognitIDE handles the study procedure automatically following the predefined workflow until after the post-questionnaire step. The workflow includes filling in questionnaires, relaxation videos, baseline recordings, and working on predefined software development tasks. After that, participants were interviewed by one of the study managers regarding their experiences during the experiment.
• Figure 2: The mapping of a person's physiological activity onto an example source code is depicted. With the help of the eye tracker and CognitIDE, it is possible to localize where the person is looking and color the source code according to heart activity, electrodermal activity, skin temperature, or brain activity.