InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs

TL;DR

InstructPipe presents a novel AI-assisted visual programming workflow for ML pipelines that uses two specialized LLM modules to generate a pipeline specification via pseudocode and a code interpreter to render the pipeline in a node-graph editor. By offloading node selection and connection to language-driven components, the system significantly reduces interactive effort and accelerates prototyping, demonstrated through technical and user evaluations with novice users. The study highlights substantial workload reductions and improved onboarding, while also revealing cognitive-load and prompting challenges that motivate future work on scalable node libraries, dynamic interactions, and responsible AI practices. Overall, InstructPipe advances human–AI collaboration for rapid, open-ended ML prototyping in visual programming environments, with practical implications for education and rapid concept exploration.

Abstract

Visual programming has the potential of providing novice programmers with a low-code experience to build customized processing pipelines. Existing systems typically require users to build pipelines from scratch, implying that novice users are expected to set up and link appropriate nodes from a blank workspace. In this paper, we introduce InstructPipe, an AI assistant for prototyping machine learning (ML) pipelines with text instructions. We contribute two large language model (LLM) modules and a code interpreter as part of our framework. The LLM modules generate pseudocode for a target pipeline, and the interpreter renders the pipeline in the node-graph editor for further human-AI collaboration. Both technical and user evaluation (N=16) shows that InstructPipe empowers users to streamline their ML pipeline workflow, reduce their learning curve, and leverage open-ended commands to spark innovative ideas.

Figures (14)

• Figure 1: The user interface of InstructPipe. The user can first click on the "InstructPipe" button on the top-right corner of the interface in (b). A dialog will appear, and the user can input the instruction and select a category tag. InstructPipe then renders a pipeline on (b), in which the user can interactively explore and revise.
• Figure 2: The distribution of 20 primitive processor nodes supported by InstructPipe. Note that "PaLM" represents two nodes in InstructPipe, i.e., a text generation model and a chat model of PaLM Anil2023PaLM.
• Figure 3: A pair example of pipeline and pseudocode. In the first line of code under "processor", pali_1_out, pali_1, pali and image=input_image_1, prompt=input_text_1 represents output variable id, node id, node type, and node arguments, respectively.
• Figure 4: The prompt structure for the Node Selection module. Each node description is formated as "{node types}: {short descriptions of the nodes}; {recommended node(s)}". The node recommendation is optional.
• Figure 5: The prompt structure for the Code Writer module. Detailed node configurations, see the appendix for examples, are listed in the highlighted region.