A State-of-the-practice Release-readiness Checklist for Generative AI-based Software Products

TL;DR

This review of gray literature identifies common challenges in deploying LLMs, from pretraining and fine-tuning to user experience considerations, from pretraining and fine-tuning to user experience considerations.

Abstract

This paper investigates the complexities of integrating Large Language Models (LLMs) into software products, with a focus on the challenges encountered for determining their readiness for release. Our systematic review of grey literature identifies common challenges in deploying LLMs, ranging from pre-training and fine-tuning to user experience considerations. The study introduces a comprehensive checklist designed to guide practitioners in evaluating key release readiness aspects such as performance, monitoring, and deployment strategies, aiming to enhance the reliability and effectiveness of LLM-based applications in real-world settings.

Figures (1)

• Figure 1: Frequency of challenges in our grey literature results (The numbers in the flowchart indicate the number of challenges associated with each stage of the LLM lifecycle)