syftr: Pareto-Optimal Generative AI

TL;DR

syftr presents a scalable framework for auto-tuning Retrieval-Augmented Generation pipelines by performing multi-objective Bayesian optimization over a massive hierarchical search space that includes agentic and non-agentic flows. It introduces a Pareto-frontier search with a novel Pareto-Pruner to efficiently balance accuracy and cost, and demonstrates across diverse RAG benchmarks that near-optimal flows can be found at a fraction of the cost of the most accurate configurations. The work shows that non-agentic flows often dominate the Pareto frontier, while upgrades to larger LLMs can yield sizable accuracy gains at substantial cost, and that transfer learning can accelerate convergence. By enabling rapid, data-driven design of high-performing generative AI pipelines, syftr has practical implications for building cost-efficient, accurate RAG systems across domains.

Abstract

Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers-each with intricate hyperparameter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early-stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average approximately 9 times cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr's ability to design and optimize allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines.

Figures (18)

• Figure 1: Given a grounding corpus, syftr searches over more than $10^{23}$ unique RAG flows to find a Pareto-frontier (optimal tradeoff curve) between task accuracy and cost.
• Figure 2: In this simplified view of the canonical RAG flow we term as "vanilla RAG", the developer has many choices for text splitter, embedding model, vector database, prompt type, and synthesizing LLM. We show a few choices to elucidate the point but there are far more choices available for each of these modules leading to hundreds of unique vanilla RAG flows each with different latency, accuracy, and cost tradeoffs.
• Figure 3: syftr RAG hierarchical search space includes $5$ top-level flows -- $4$ agentic and $1$ non-agentic with a total of $10^{23}$ unique flows. The agentic flows use the RAG flow as a subroutine while adding their own unique hyperparameters.
• Figure 4: Pareto-Pruner estimates confidence intervals around task accuracy and cost for a given flow, and will early-terminate flows whose upper-left confidence point ($p$) falls below the current Pareto-frontier. $c$ is the P80 cost, $a$ is the average accuracy, $L$ is the number of successful evaluations, $N$ is the number of total evaluations including errors caused by issues like content filtering, endpoint rate limit hits and agentic flow failures due to improper tool usage. $\sigma_c$ is the standard deviation of the costs of the current evaluation, and $z$ is the standard score for a normal distribution and controls the sensitivity of the Pareto-Pruner.
• Figure 5: Multi-Dataset Study: Pareto-frontier for InfiniteBench; See Appendix \ref{['sec:multi_dataset_study_details']} for other datasets. Colored dots represent flows whose key components are described in the legend. Legend items are sorted by descending accuracy, except for the flow denoted Baseline at the bottom, which is a baseline RAG flow that is similar to LlamaIndex default settings and uses gpt-4o-mini as the synthesizing LLM and bge-small-en-v1.5 embeddings. Note the x-axis is log scale.