PromptChainer: Chaining Large Language Model Prompts through Visual Programming

TL;DR

The paper tackles the challenge that single LLM prompts cannot handle complex, multi-step tasks. It introduces PromptChainer, a visual programming interface with chain and node views for authoring LLM chains, plus debugging tools. Through formative studies with four designers/developers, it identifies key challenges like output transformation, unstable function signatures, and cascading errors, and shows how the tool supports multi-node chains and debugging. It also raises open questions about scaling chains to more complex tasks and enabling rapid, low-fi prototyping of alternative chain structures.

Abstract

While LLMs can effectively help prototype single ML functionalities, many real-world applications involve complex tasks that cannot be easily handled via a single run of an LLM. Recent work has found that chaining multiple LLM runs together (with the output of one step being the input to the next) can help users accomplish these more complex tasks, and in a way that is perceived to be more transparent and controllable. However, it remains unknown what users need when authoring their own LLM chains -- a key step for lowering the barriers for non-AI-experts to prototype AI-infused applications. In this work, we explore the LLM chain authoring process. We conclude from pilot studies find that chaining requires careful scaffolding for transforming intermediate node outputs, as well as debugging the chain at multiple granularities; to help with these needs, we designed PromptChainer, an interactive interface for visually programming chains. Through case studies with four people, we show that PromptChainer supports building prototypes for a range of applications, and conclude with open questions on scaling chains to complex tasks, and supporting low-fi chain prototyping.

Figures (6)

• Figure 1: The PromptChainer interface. (A) The Chain View visualizes the chain structure with node-edge diagrams (enlarged in Figure \ref{['fig:example']}), and allows users to edit the chain by adding, removing, or reconnecting nodes. (B) The Node View supports implementing, improving, and testing each individual node, e.g., editing prompts for LLM nodes. PromptChainer also supports running the chain end-to-end (C).
• Figure 2: An example chain for prototyping music chatbot, modified from a pilot user's chain (its overview is in Figure \ref{['fig:ui']}). We provide primary input-output examples, and annotate the node functionalities are annotated inline.
• Figure 4: An expansion of Figure \ref{['fig:example']}, is about music: (A) Node visualization: the node has an status icon ($a_1$), a list of named input ($a_2$) and output handles ($a_3$), as well as detailed data previews ($a_4$). (B) Implementation: the handle names are synchronized with the underlying prompt template ($b_1$). (C) We can debug the node at multiple levels.
• Figure 5: Four different chains built by user study participants. P1 and P2's chains used parallel branching logic , whereas P3 and P4's chains depict iterative content processing. The full details are in Figure \ref{['fig:user_chains_full']}, Appendix \ref{['appendix:pilot_chains']}.
• Figure 6: The full details of user study chains.