Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows

TL;DR

The paper tackles the challenge of designing effective LLM chains by borrowing structured workflows from crowdsourcing. It builds a design space categorized by objectives (quality, objectivity, generality) and tactics (actors, subtasks, architectures), grounded in a systematic review of 107 papers, and validates the space through three case studies that adapt crowdsourcing workflows to LLM chains. The results reveal that many crowdsourcing strategies transfer to LLM chains, but require task-specific, model-aware adaptations in subtasks, validation, and context handling, with notable tradeoffs in cost and latency. The work offers practical guidance for chain design, highlights opportunities for user-in-the-loop approaches and subjective tasks, and calls for tooling to support systematic chain design as LLM capabilities evolve.

Abstract

LLM chains enable complex tasks by decomposing work into a sequence of subtasks. Similarly, the more established techniques of crowdsourcing workflows decompose complex tasks into smaller tasks for human crowdworkers. Chains address LLM errors analogously to the way crowdsourcing workflows address human error. To characterize opportunities for LLM chaining, we survey 107 papers across the crowdsourcing and chaining literature to construct a design space for chain development. The design space covers a designer's objectives and the tactics used to build workflows. We then surface strategies that mediate how workflows use tactics to achieve objectives. To explore how techniques from crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to implement LLM chains across three case studies: creating a taxonomy, shortening text, and writing a short story. From the design space and our case studies, we identify takeaways for effective chain design and raise implications for future research and development.

Figures (19)

• Figure 1: Objectives. Our design space contains three axes of workflow objectives, shown as rows above. The first regards the impact of resource constraints on the ability to improve outcome quality. The second axis distinguishes different notions of 'quality:' whether the task objective requires objectivity, subjectivity, or elements of both. The final axis concerns generality: if the workflow is aimed to solve a single, specific task or to generalize to many tasks.
• Figure 2: Tactics. Tactics are the building blocks of workflows. Actors complete subtasks, which are connected together in architectures. There are multiple categories of actors (crowdworkers, LLMs, and the user), subtasks (generate, evaluate, improve, focus, partition and merge), and architectures (sequential, branching, redundant, dynamic, and communicative). Workflows may use multiple tactics in any of these categories.
• Figure 3: The distribution of the corpus in the design space. We categorize the papers in our corpus across different categories of the design space. Each set of bars is the categorization, split by the crowdsourcing and LLM literatures. There are fewer LLM papers in our corpus (39) than crowdsourcing papers (68). The length of the bar is the proportion of all papers in each literature that incorporate the design space element. We provide the categorization of each paper, as well as the criteria for our categorizations in the supplementary materials. Through this comparison, we see how the literatures similarly span the entirety of the design space, with the exception of using LLMs or crowdworkers. We surface some differences as well, such as how crowdsourcing papers are more often task-specific, incorporate subjectivity, and use communicative architectures.
• Figure 4: Design space connections. In working towards objectives, challenges can arise. This figure captures a subset of possible issues in achieving objectives. Based on our design space, these issues may be addressed by utilizing different strategies. In turn, these strategies are supported by various tactics. For example, if the objective requires navigating a large creative space of subjective outcomes, a challenging proposition, then designers can incorporate the adaptive architectures, diverse responses, and user guidance strategies. Each of these strategies has been implemented in prior work by a variety of architectures and other tactics.
• Figure 5: An overview of the tactics used for and the effects of the diverse responses strategy.