Themes of Building LLM-based Applications for Production: A Practitioner's View

TL;DR

This study maps practitioner-facing themes for building LLM-based applications in production by analyzing 189 publicly available YouTube talks. Using semi-automated topic modeling (BERTopic) and manual refinement, it identifies eight themes and 20 topics, with Architecture & Design and RAG systems as central concerns. Key contributions include a thematic map of practitioner discussions, insights into co-occurrence patterns, and a replication package linking videos to topics, highlighting practical challenges in architecture, data handling, evaluation, security, and costs. The work offers a bridge between industry practice and research, guiding practitioners in LLM deployment and suggesting future research directions around architectural decisions and production-specific hyperparameter tuning.

Abstract

Background: Large language models (LLMs) have become a paramount interest of researchers and practitioners alike, yet a comprehensive overview of key considerations for those developing LLM-based systems is lacking. This study addresses this gap by collecting and mapping the topics practitioners discuss online, offering practical insights into where priorities lie in developing LLM-based applications. Method: We collected 189 videos from 2022 to 2024 from practitioners actively developing such systems and discussing various aspects they encounter during development and deployment of LLMs in production. We analyzed the transcripts using BERTopic, then manually sorted and merged the generated topics into themes, leading to a total of 20 topics in 8 themes. Results: The most prevalent topics fall within the theme Design & Architecture, with a strong focus on retrieval-augmented generation (RAG) systems. Other frequently discussed topics include model capabilities and enhancement techniques (e.g., fine-tuning, prompt engineering), infrastructure and tooling, and risks and ethical challenges. Implications: Our results highlight current discussions and challenges in deploying LLMs in production. This way, we provide a systematic overview of key aspects practitioners should be aware of when developing LLM-based applications. We further pale off topics of interest for academics where further research is needed.

Figures (3)

• Figure 1: Overview of the different steps of our applied methodology.
• Figure 2: Themes and topics practitioners discuss with regard to LLMs in Production. The size of a box encodes its topic frequency over all videos.
• Figure 3: Co-occurrences of topics where darker colors highlight topics occur more often together within individual videos.