Comprehension vs. Adoption: Evaluating a Language Workbench Through a Family of Experiments

TL;DR

This study investigates Neverlang, a modular language workbench, through a family of three quasi-experiments that adapt the Method Evaluation Model to assess both comprehensibility of its meta-language and user acceptance. Across replications, learners showed substantial comprehension, especially of syntax, while perceived usefulness and intention to use were favorable, but perceived ease of use remained a notable challenge. Meta-analyses reveal a consistent gap between high comprehension (mean around $0.69$ overall) and modest ease of use (around $0.48$), with acceptance driven by usefulness and ease of use influencing intention to adopt. The work provides a structured framework for evaluating LWBs from a user-centered perspective and highlights actionable directions for improving documentation, libraries, and interfaces to enhance adoption in educational and research settings.

Abstract

Language workbenches are tools that enable the definition, reuse, and composition of programming languages and their ecosystems, aiming to streamline language development. To facilitate their adoption by language designers, the comprehensibility of the language used to define other languages is an important aspect to evaluate. Moreover, considering that language workbenches are relatively new tools, user acceptance emerges as a crucial factor to be accounted for during their assessment. Current literature often neglects user-centred aspects like comprehensibility and acceptance in the assessment of this breed of tools. This paper addresses this gap through a family of experiments assessing Neverlang, a modular language workbench. The study adopts a tailored version of the Method Evaluation Model (MEM) to evaluate the comprehensibility of Neverlang's meta-language and programs, as well as user acceptance in terms of perceived ease of use, perceived usefulness, and intention to use. It also investigates the relationships among these dimensions. The experiments were conducted in three iterations involving participants from academia. The results reveal that users demonstrate sufficient comprehension of Neverlang's meta-language, particularly concerning its syntax, express a favourable perception of its usefulness, and indicate their intention to use it. However, the results also indicate that Neverlang's ease of use remains a challenge. Additionally, variations in the perceived ease of use and perceived usefulness, whether low or high, influence the users' intention to use the tool. Surprisingly, no significant correlation is found between comprehensibility and user acceptance. Notably, higher comprehensibility of the meta-language does not necessarily translate into greater acceptance, underscoring the complex interplay between comprehension and adoption.

Figures (5)

• Figure 1: Exemplary language fragment in Neverlang.
• Figure 2: Tailored MEM. Actual success is expected to impact perceived success. Actual success is used as a proxy for comprehensibility, which is further decomposed into learnability, understandability, and evolving. Perceived success is decomposed into perceived ease of use, perceived usefulness, and intention to use.
• Figure 3: Example of a test question.
• Figure 4: Self-evaluated participants' prior knowledge ranging from 1 (very low) to 5 (very high).
• Figure 5: Forest plots summarizing the results of the meta-analyses of proportions across the three experiments for each measured dimension.