Exploring Influence Factors on LLM Suitability for No-Code Development of End User IoT Applications

TL;DR

This work investigates how four factors—model selection, prompt language, training data background, and error-informed few-shot prompts—affect the suitability of Large Language Models for end-user no-code development of IoT applications. Using a representative LLM-powered no-code platform (LLM4FaaS) and a set of eight models across four real smart-home tasks, the study provides a nuanced, cross-model evaluation of syntactic and semantic accuracy, including cross-language (Chinese vs English) and few-shot vs zero-shot settings. Key findings show that general-purpose LLMs (e.g., GPT-4o) generally outperform coding-focused ones for end-user tasks, language alignment with the input can influence performance, and error-informed few-shot prompts can yield meaningful gains, especially for mid-range models. The results offer practical guidance for platform designers on model selection, prompt design, and the viable use of iterative feedback to improve executable, user-aligned automation in no-code IoT contexts.

Abstract

No-Code Development Platforms (NCDPs) empower non-technical end users to build applications tailored to their specific demands without writing code. While NCDPs lower technical barriers, users still require some technical knowledge, e.g., to structure process steps or define event-action rules. Large Language Models (LLMs) offer a promising solution to further reduce technical requirements by supporting natural language interaction and dynamic code generation. By integrating LLM, NCDPs can be more accessible to non-technical users, enabling application development truly without requiring any technical expertise. Despite growing interest in LLM-powered NCDPs, a systematic investigation into the factors influencing LLM suitability and performance remains absent. Understanding these factors is critical to effectively leveraging LLMs capabilities and maximizing their impact. In this paper, we investigate key factors influencing the effectiveness of LLMs in supporting end-user application development within NCDPs. By conducting comprehensive experiments, we evaluate the impact of four key factors, i.e., model selection, prompt language, training data background, and an error-informed few-shot setup, on the quality of generated applications. Specifically, we selected a range of LLMs based on their architecture, scale, design focus, and training data, and evaluated them across four real-world smart home automation scenarios implemented on a representative open-source LLM-powered NCDP. Our findings offer practical insights into how LLMs can be effectively integrated into NCDPs, informing both platform design and the selection of suitable LLMs for end-user application development.

Figures (16)

• Figure 1: User Response to Tasks of Varying Complexity: This figure presents a real user response from the dataset, illustrating four smart home automation tasks across different complexity levels. The response showcases increasing complexity, from simple device control to intricate automation involving real-time sensor data and user-environment coordination. The original response is in Chinese, and we provide an English translation for clarity.
• Figure 2: Dataset Example: We use a LLM-powered No-Code Platform, i.e., LLM4FaaS, as the base NCDP for evaluation. Specifically, it takes the user natural language description in combined with the project context and reference code as input to LLMs. To evaluate the accuracy of LLM-generated results, we set the ground truth based on the user requirements. The accuracy is set by comparing the output from generated results and the ground truth. Essentially, the dataset consists of user requirements and the corresponding ground truth.
• Figure 3: We consider syntactic success as error-free results and semantic success as results fully meets user requirements. The graphs depict results based on Chinese user prompts, the original language of the used dataset. GPT-4o shows distinct advantages in both syntactic and semantic success compared to other models. GPT-4o-mini performs adequately on the easy task, but the performance drops significantly on more complex ones.
• Figure 4: GPT-4o consistently outperforms other models in semantic accuracy across tasks of varying difficulty, showing a higher accuracy rate in cases of non-semantic-success. As the task complexity increases, the semantic accuracy distribution of results becomes more dispersed. The results show a bimodal distribution, with most tasks either failing completely or achieving 100% success.
• Figure 5: Success rates with English prompts. GPT-4o still outperforms other models with English prompts in both syntactic and semantic success.