Software Performance Engineering for Foundation Model-Powered Software

TL;DR

FMware introduces production-level performance guarantees for foundation-model-powered software by treating performance engineering as a first-class concern. The authors identify four SPE challenges—cognitive-architecture design, token-efficient communication, tuning/optimization, and deployment—and propose the SLA-aware FMware Runtime with Profiler, Resource Provisioner, and Replica Router to enforce end-to-end SLAs using a latency estimate $l_{est} = l_x + l_q + l_r$. Their evaluation against Ray Serve on a heterogeneous cluster demonstrates substantial latency and goodput improvements, validating the approach and highlighting gains in resource utilization. The work advances practical, SLA-driven engineering for FMware and calls for benchmarks and tooling to broaden adoption across Promptware and Agentware projects.

Abstract

The rise of Foundation Models (FMs) like Large Language Models (LLMs) is revolutionizing software development. Despite the impressive prototypes, transforming FMware into production-ready products demands complex engineering across various domains. A critical but overlooked aspect is performance engineering, which aims at ensuring FMware meets performance goals such as throughput and latency to avoid user dissatisfaction and financial loss. Often, performance considerations are an afterthought, leading to costly optimization efforts post-deployment. FMware's high computational resource demands highlight the need for efficient hardware use. Continuous performance engineering is essential to prevent degradation. This paper highlights the significance of Software Performance Engineering (SPE) in FMware, identifying four key challenges: cognitive architecture design (i.e., the structural design that defines how AI components interact, reason, and interface with classical software components), communication protocols, tuning and optimization, and deployment. These challenges are based on literature surveys and experiences from developing an in-house FMware system. We discuss problems, current practices, and innovative paths for the software engineering community.

Figures (3)

• Figure 1: The illustration of software performance engineering challenges for FMware throughout the development lifecycle
• Figure 2: A flowchart of the RAG-based LLM pipeline from OPEA opea_image, with added annotations on the right panel to highlight key tuning parameters and decision points, illustrating the complexities involved in FMware development
• Figure 3: The simplified architecture and components of FMware Runtime