Can a domain-specific language improve program structure comprehension of data pipelines? A mixed-methods study

TL;DR

The study evaluates whether Jayvee, a domain-specific language for data pipelines, can improve bottom-up program structure comprehension relative to Python/Pandas. Using a mixed-methods design with a controlled crossover experiment (57 masters-level participants) and a descriptive survey, the authors find no speed advantage for Jayvee but a significant improvement in correctness when understanding data pipelines. Qualitative analysis reveals that pipeline-overview, enforced structure, and alignment with the data-pipeline mental model contribute to better comprehension, while aspects like high density of functionality, available options, and unfamiliar syntax can hinder understanding for some readers. The results suggest DSLs hold promise for enabling domain experts to contribute more accurately to data pipelines, though open questions about abstraction level, scalability, and library ecosystems remain for future work.

Abstract

In many application domains, domain-specific languages can allow domain experts to contribute to collaborative projects more correctly and efficiently. To do so, they must be able to understand program structure from reading existing source code. With high-quality data becoming an increasingly important resource, the creation of data pipelines is an important application domain for domain-specific languages. We execute a mixed-method study consisting of a controlled experiment and a follow-up descriptive survey among the participants to understand the effects of a domain-specific language on bottom-up program understanding and generate hypotheses for future research. During the experiment, participants need the same time to solve program structure comprehension tasks, but are significantly more correct when using the domain-specific language. In the descriptive survey, participants describe reasons related to the programming language itself, such as a better pipeline overview, more enforced code structure, and a closer alignment to the mental model of a data pipeline. In addition, human factors such as less required programming experience and the ability to reuse experience from other data engineering tools are discussed. Based on these results, domain-specific languages are a promising tool for creating data pipelines that can increase correct understanding of program structure and lower barriers to entry for domain experts. Open questions exist to make more informed implementation decisions for domain-specific languages for data pipelines in the future.

Figures (10)

• Figure 1: Overview of the mixed method research design, split into data collection and data analysis.
• Figure 2: The experiment tool during task 2 in Python/Pandas. Pipeline source code is shown on the left, the recreation using ordered steps on the right.
• Figure 3: Comparison of source code excerpts to extract data from a CSV source, shown for task 1 in Jayvee and Python/Pandas.
• Figure 4: Kernel-density-plot of correctness of solution for Jayvee compared to Python/Pandas.
• Figure 5: Code assignments compared to codebook changes during thematic analysis, showing codebook changes being rare after the third iteration, while codes were consistently applied to new responses.