On the impact of semantic transparency on understanding and reviewing social goal models

TL;DR

This study investigates whether increasing semantic transparency in the i* concrete syntax improves stakeholders' ability to understand and review i* SR models. Using a quasi-experiment with 57 novice participants, the authors compare standard i* notation to a semantically transparent variant in the presence of a language key, assessing accuracy, speed, and ease via task performance and eye-tracking data. They find no improvement in accuracy or speed with the transparent syntax, though participants exhibit reduced visual effort, suggesting context and the language key mitigate symbol-recognition deficits. The results imply that semantic transparency alone may not enhance model comprehension in practice, and highlight the importance of evaluating multiple PoN principles and real-world contexts in notation design.

Abstract

Context: i* is one of the most influential languages in the Requirements Engineering research community. Perhaps due to its complexity and low adoption in industry, it became a natural candidate for studies aiming at improving its concrete syntax and the stakeholders' ability to correctly interpret i* models. Objectives: We evaluate the impact of semantic transparency on understanding and reviewing i* models, in the presence of a language key. Methods: We performed a quasi-experiment comparing the standard i* concrete syntax with an alternative that has an increased semantic transparency. We asked 57 novice participants to perform understanding and reviewing tasks on i* models, and measured their accuracy, speed and ease, using metrics of task success, time and effort, collected with eye-tracking and participants' feedback. Results: We found no evidence of improved accuracy or speed attributable to the alternative concrete syntax. Although participants' perceived ease was similar, they devoted significantly less visual effort to the model and the provided language key, when using the alternative concrete syntax. Conclusions: The context provided by the model and language key may mitigate the i* symbol recognition deficit reported in previous works. However, the alternative concrete syntax required a significantly lower visual effort.

Figures (4)

• Figure 1: Standard i* symbol set
• Figure 2: New i* symbol set Caire2013RE
• Figure 3: Understand and review tasks proposed to participants
• Figure 4: Heat maps for the understand and review tasks in both notations