An Empirical Study on Low-Code Programming using Traditional vs Large Language Model Support

TL;DR

This paper empirically contrasts traditional low-code programming (LCP) with LLM-based LCP by analyzing Stack Overflow discussions over three years to map application domains, software development tasks, and limitations. It employs a three-step methodology—data collection from Stack Overflow, ChatGPT-assisted data extraction, and constant-comparison qualitative analysis—to build datasets of 411 traditional and 595 LLM-based LCP posts, uncovering broader domain coverage and lifecycle differences for LLM-based LCP. Key findings show that LLM-based LCP spans more domains and emphasizes implementation and deployment, but faces reliability, prompt guidance, and hallucination challenges, with a relatively modest positive feedback rate. The work offers actionable insights for tool developers and proposes data sharing and multi-source triangulation for future research on LCP with and without LLM support.

Abstract

Low-code programming (LCP) refers to programming using models at higher levels of abstraction, resulting in less manual and more efficient programming, and reduced learning effort for amateur developers. Many LCP tools have rapidly evolved and have benefited from the concepts of visual programming languages (VPLs) and programming by demonstration (PBD). With the huge increase in interest in using large language models (LLMs) in software engineering, LLM-based LCP has begun to become increasingly important. However, the technical principles and application scenarios of traditional approaches to LCP and LLM-based LCP are significantly different. Understanding these key differences and characteristics in the application of the two approaches to LCP by users is crucial for LCP providers in improving existing and developing new LCP tools and in better assisting users in choosing the appropriate LCP technology. We conducted an empirical study of both traditional LCP and LLM-based LCP. We analyzed developers' discussions on Stack Overflow (SO) over the past three years and then explored the similarities and differences between traditional LCP and LLM-based LCP features and developer feedback. Our findings reveal that while traditional LCP and LLM-based LCP share common primary usage scenarios, they significantly differ in scope, limitations, and usage throughout the software development lifecycle, particularly during the implementation phase. We also examine how LLMs impact and integrate with LCP, discussing the latest technological developments in LLM-based LCP, such as its integration with VPLs and the application of LLM Agents in software engineering.

Figures (1)

• Figure 1: Overview of the investigation process, each step corresponds to a subsection in Section \ref{['Methodology']}