Task Cascades for Efficient Unstructured Data Processing

TL;DR

Task Cascades generalize model cascades by allowing each processing stage to choose a (potentially surrogate) operation, a model, and a document fraction. The authors introduce a greedy, agent-driven method to assemble cost-efficient cascades that meet a user-specified accuracy target with statistical guarantees, including document restructuring and surrogate generation components. Across eight real-world document classification tasks, Task Cascades achieve substantial end-to-end cost reductions (average ≈48% vs model cascades and ≈86% vs oracle) while maintaining target accuracy, with robust performance across varying target levels and workloads. The approach is practical at production scales, with a favorable break-even profile and scalable offline optimization, suggesting strong potential for deploying cheaper, scalable LLM-powered text processing in real systems.

Abstract

Modern database systems allow users to query or process unstructured text or document columns using LLM-powered functions. Users can express an operation in natural language (e.g., "identify if this review mentions billing issues"), with the system executing the operation on each document, in a row-by-row fashion. One way to reduce cost on a batch of documents is to employ the model cascade framework: a cheap proxy model processes each document, and only uncertain cases are escalated to a more accurate, expensive oracle. However, model cascades miss important optimization opportunities; for example, often only part of a document is needed to answer a query, or other related, but simpler operations (e.g., "is the review sentiment negative?", "does the review mention money?") can be handled by cheap models more effectively than the original operation, while still being correlated with it. We introduce the task cascades framework, which generalizes model cascades by varying not just the model, but also the document portion and operation at each stage. Our framework uses an LLM agent to generate simplified, decomposed, or otherwise related operations and selects the most relevant document portions, constructing hundreds of candidate tasks from which it assembles a task cascade. We show that optimal cascade selection is intractable via reduction from Minimum Sum Set Cover, but our iterative approach constructs effective cascades. We also provide an extension that offers statistical accuracy guarantees: the resulting cascade meets a user-defined accuracy target (with respect to the oracle) up to a bounded failure probability. Across eight real-world document processing tasks at a 90% target accuracy, task cascades reduce end-to-end cost by an average of 36% compared to model cascades, at a production scale.

Figures (8)

• Figure 1: Task vs. model cascade for determining whether a Supreme Court opinion overturns a lower court decision in \ref{['ex:supreme-court-task']}. In a task cascade (left), each stage (or task) is defined by a document fraction ($f$), a model ($m$), and an operation ($o$). Surrogate operations may be reused at different document fractions (as in tasks 1 and 3). The final oracle task applies the original user-specified operation ($o^*$) on the full document ($f=1.0$) with the oracle model ($m=$gpt-4o) if prior tasks cannot confidently resolve the input. In contrast, a model cascade (right) applies the same operation $o^*$ to the full document at each stage, varying only the model.
• Figure 2: Overview of our approach. A user poses a complex classification query over long documents. Our approach restructures each document to prioritize relevant chunks, then iteratively proposes surrogate operations and assembles a cost-effective task cascade to meet accuracy constraints.
• Figure 3: Standardizing documents. This format allows the oracle to specify relevant content as line ranges.
• Figure 4: Cost stability of 2-Model Cascade (+ Guarantees) and Task Cascades (+ Guarantees) across 10 independent runs for five representative workloads. Each box shows the distribution of total inference cost; red dashed lines indicate the median. Both methods exhibit high variance.
• Figure 5: Accuracy vs. cost trade-offs as the target accuracy varies 75% to 95% (in increments of 5%). Task cascades (green) dominate the Pareto frontier on easy workloads and provide robust gains or new operating points on harder tasks. On simple workloads like agnews, cost improvements appear mainly at lower targets, where the baselines cannot match. LOTUS (+ Guarantees) (pink) only supports binary classification workloads, so we exclude agnews.

Theorems & Definitions (3)

• theorem 1
• theorem 2
• lemma 1: Corollary to \ref{['thm:guarantees']} by waudby2024estimating