LangProBe: a Language Programs Benchmark

TL;DR

LangProBe addresses the challenge of evaluating modular language-program pipelines by examining cost–quality tradeoffs across architectures, datasets, optimizers, and language models. It introduces a large-scale benchmark with 15 datasets, multiple program designs, and four optimizers, enabling over 2000 experiments to map Pareto frontiers. The study finds that optimized language programs can yield cost–quality improvements over raw calls to stronger baselines, but performance remains task-dependent and requires careful design and empirical selection. By providing open-source data and a flexible evaluation framework, LangProBe offers a practical platform for advancing modular AI systems and guiding practitioners toward cost-efficient, high-quality deployments.

Abstract

Composing language models (LMs) into multi-step language programs and automatically optimizing their modular prompts is now a mainstream paradigm for building AI systems, but the tradeoffs in this space have only scarcely been studied before. We introduce LangProBe, the first large-scale benchmark for evaluating the architectures and optimization strategies for language programs, with over 2000 combinations of tasks, architectures, optimizers, and choices of LMs. Using LangProBe, we are the first to study the impact of program architectures and optimizers (and their compositions together and with different models) on tradeoffs of quality and cost. We find that optimized language programs offer strong cost--quality Pareto improvement over raw calls to models, but simultaneously demonstrate that human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance. We will open source the code and evaluation data for LangProBe.

Figures (10)

• Figure 1: LangProBe includes 15 datasets, 4 optimizers, and more than 10 language programs, creating more than 400 configurations for evaluating language programs with different tasks and optimizers.
• Figure 2: This figure shows stark cost-performance trade-offs across various configurations of (LM, Language Program, Optimizer), aggregated over multiple datasets in LangProBe. The Pareto curves represent the upper-left convex hull of achievable configurations. Piece-wise linear Pareto curve segments appear curved due to the log scale, but all points on the Pareto front are achievable via weighted (randomized) choice between the two endpoints. Four configurations are compared: 1) Model: Performance of baseline program (e.g., raw model predictions) without optimizers. 2) Model+Program: Performance with language programs applied, without optimizers. 3) Model+Optimizer: Performance with optimizers applied to the baseline program. 4) Model+Program+Optimizer: Performance of combined use of both language programs and optimizers. Key Takeaway: For both model families, the Model+Program+Optimizer Pareto curve deliver cost and quality improvements against Model+Program and Model+Optimizer Pareto curves, which in turn improve over the Model Pareto curve, implying that using language programs and optimizing them can offer considerable gains not only with respect to quality, but also cost.
• Figure 3: Performance comparison between best-performing programs and the baseline. In most cases, the baseline is a zero-shot call to the LM with task description and task inputs. Scores are averaged from all language models we evaluated, including both OpenAI models and Llama models. In almost all tasks, both optimized and unoptimized programs perform better than the raw model prediction baseline.
• Figure 4: Performance comparison to Baseline, with best performing and worst performing programs for all Knowledge, Reasoning, and Math tasks. All programs are unoptimized. On the same dataset, different language programs' performances vary. Similarly, the same language program's performance varies on different datasets.
• Figure 5: Frequency Distribution for individual optimizer performance, ranked by the number of times that an optimizer applied on a program is within 3% of that program's highest score (blue bar). We also note the number of the highest-performing optimizer as the top score (green bar). From the plot, MIPROv2-T, which uses a stronger model for optimization to propose better instructions combined with corresponding few-shot examples through Bayesian search, works the best.