Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cycle

Jens Kohl, Luisa Gloger, Rui Costa, Otto Kruse, Manuel P. Luitz, David Katz, Gonzalo Barbeito, Markus Schweier, Ryan French, Jonas Schroeder, Thomas Riedl, Raphael Perri, Youssef Mostafa

TL;DR

The paper presents the Generative AI Toolkit, an open-source framework that automates the full life cycle of LLM-based applications, addressing labor-intensive DevOps tasks from agent creation to runtime monitoring. By enabling custom metrics, repeatable evaluation cases, model/prompt comparisons, automated tests, infrastructure-as-code deployment, CI/CD integration, a debugging GUI, and scalable traces, the toolkit aims to improve quality and reduce release cycles for AI-powered agents. The authors validate the approach with multi-use-case demonstrations and discuss best practices, while acknowledging that quantifying universal gains depends on context. The work emphasizes practical impact by offering a ready-to-use, end-to-end solution that teams can adopt, adapt, and extend as open source.

Abstract

As LLM-based applications reach millions of customers, ensuring their scalability and continuous quality improvement is critical for success. However, the current workflows for developing, maintaining, and operating (DevOps) these applications are predominantly manual, slow, and based on trial-and-error. With this paper we introduce the Generative AI Toolkit, which automates essential workflows over the whole life cycle of LLM-based applications. The toolkit helps to configure, test, continuously monitor and optimize Generative AI applications such as agents, thus significantly improving quality while shortening release cycles. We showcase the effectiveness of our toolkit on representative use cases, share best practices, and outline future enhancements. Since we are convinced that our Generative AI Toolkit is helpful for other teams, we are open sourcing it on and hope that others will use, forward, adapt and improve

Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cycle

TL;DR

The paper presents the Generative AI Toolkit, an open-source framework that automates the full life cycle of LLM-based applications, addressing labor-intensive DevOps tasks from agent creation to runtime monitoring. By enabling custom metrics, repeatable evaluation cases, model/prompt comparisons, automated tests, infrastructure-as-code deployment, CI/CD integration, a debugging GUI, and scalable traces, the toolkit aims to improve quality and reduce release cycles for AI-powered agents. The authors validate the approach with multi-use-case demonstrations and discuss best practices, while acknowledging that quantifying universal gains depends on context. The work emphasizes practical impact by offering a ready-to-use, end-to-end solution that teams can adopt, adapt, and extend as open source.

Abstract

As LLM-based applications reach millions of customers, ensuring their scalability and continuous quality improvement is critical for success. However, the current workflows for developing, maintaining, and operating (DevOps) these applications are predominantly manual, slow, and based on trial-and-error. With this paper we introduce the Generative AI Toolkit, which automates essential workflows over the whole life cycle of LLM-based applications. The toolkit helps to configure, test, continuously monitor and optimize Generative AI applications such as agents, thus significantly improving quality while shortening release cycles. We showcase the effectiveness of our toolkit on representative use cases, share best practices, and outline future enhancements. Since we are convinced that our Generative AI Toolkit is helpful for other teams, we are open sourcing it on and hope that others will use, forward, adapt and improve

Paper Structure

This paper contains 33 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Challenges in developing and operating LLM-based applications over their whole life cycle
  3. Overview of our contribution
  4. Related Work
  5. DevOps:
  6. Continuous Integration and Continuous Delivery (CI/CD):
  7. Software testing:
  8. Large-language models (LLM):
  9. LLM-based agents:
  10. Metrics for LLM-based applications:
  11. Testing LLM-based applications:
  12. Machine Learning and LLM operations:
  13. LLM-frameworks:
  14. Contribution
  15. Overview of our contributions
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Overview Generative AI Toolkit and its features.
  • Figure 2: Screenshot of Generative AI Toolkit's CI/CD pipeline.
  • Figure 3: Screenshot of Generative AI Toolkit’s GUI for "debugging" agents.
  • Figure 4: Screenshot of Amazon CloudWatch metric for agent after deployment.
  • Figure 5: Screenshot of test results for use case 1
  • ...and 1 more figures