SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines

TL;DR

SPADE automates the generation and filtering of data quality assertions for large language model pipelines by mining prompt version histories and synthesizing assertion criteria from prompt deltas. It deploys a two phase approach: first, a candidate assertion generator builds a multiset of Python based assertions from delta categories, and second, an ILP based filter selects a minimal, high coverage, low false failure set potentially augmented with a subsumption mechanism for data sparse settings. The framework is validated on nine real world pipelines, showing reductions in the number of assertions and false failures, and is deployed within LangSmith to support thousands of pipelines across industries. The work advances robust LLM pipeline deployment by providing automated, scalable data quality checks with practical tooling and public data and code resources.

Abstract

Large language models (LLMs) are being increasingly deployed as part of pipelines that repeatedly process or generate data of some sort. However, a common barrier to deployment are the frequent and often unpredictable errors that plague LLMs. Acknowledging the inevitability of these errors, we propose {\em data quality assertions} to identify when LLMs may be making mistakes. We present SPADE, a method for automatically synthesizing data quality assertions that identify bad LLM outputs. We make the observation that developers often identify data quality issues during prototyping prior to deployment, and attempt to address them by adding instructions to the LLM prompt over time. SPADE therefore analyzes histories of prompt versions over time to create candidate assertion functions and then selects a minimal set that fulfills both coverage and accuracy requirements. In testing across nine different real-world LLM pipelines, SPADE efficiently reduces the number of assertions by 14\% and decreases false failures by 21\% when compared to simpler baselines. SPADE has been deployed as an offering within LangSmith, LangChain's LLM pipeline hub, and has been used to generate data quality assertions for over 2000 pipelines across a spectrum of industries.

Figures (6)

• Figure 1: Before a developer deploys a prompt template to production, spade analyzes the deltas (i.e., diffs) between consecutive prompt templates to generate assertions. Then, spade uses labeled pipeline inputs and outputs to filter out redundant and inaccurate assertions, while maintaining coverage of bad outputs.
• Figure 2: Taxonomy of prompt deltas from 19 LLM pipelines.
• Figure 3: Generating candidate assertions from a $\Delta\mathcal{P}$.
• Figure 4: Screenshot of an early version of the spade Streamlit application.
• Figure 5: Prompts submitted to the spade Streamlit application span a variety of fields.

Theorems & Definitions (3)

• definition 1
• definition 2
• definition 3