What Slows Down FMware Development? An Empirical Study of Developer Challenges and Resolution Times
Zitao Wang, Zhimin Zhao, Michael W. Godfrey
TL;DR
This study provides the first large-scale empirical analysis of FMware development by examining FMware deployed on cloud Promptware platforms and on GitHub-hosted repositories. Using BERTopic-based topic modeling on 25,790 Promptware descriptions and 146,570 GitHub issues, the authors generate a taxonomy of 18 macrotopics and 15- to 30-topic categorizations that reveal dominant domains (education, content creation, business) and persistent technical and operational challenges (memory management, dependencies, tokenizer config, and prompt design). Time-to-resolve analyses show that issues related to code contribution, similarity search, and prompt templates demand the most developer effort, while infrastructure and model-training topics are most time-consuming in core functionality. The findings offer actionable guidance for tooling, debugging support, and cross-platform workflows, informing researchers and tool builders on where to target improvements to accelerate robust FMware development across platforms and deployment models.
Abstract
Foundation Models (FMs), such as OpenAI's GPT, are fundamentally transforming the practice of software engineering by enabling the development of \emph{FMware} -- applications and infrastructures built around these models. FMware systems now support tasks such as code generation, natural-language interaction, knowledge integration, and multi-modal content creation, underscoring their disruptive impact on current software engineering workflows. However, the design, implementation, and evolution of FMware present significant new challenges, particularly across cloud-based and on-premise platforms where goals, processes, and tools often diverge from those of traditional software development. To our knowledge, this is the first large-scale analysis of FMware development across both cloud-based platforms and open-source repositories. We empirically investigate the FMware ecosystem through three focus areas: (1) the most common application domains of FMware, (2) the key challenges developers encounter, and (3) the types of issues that demand the greatest effort to resolve. Our analysis draws on data from GitHub repositories and from leading FMware platforms, including HuggingFace, GPTStore, Ora, and Poe. Our findings reveal a strong focus on education, content creation, and business strategy, alongside persistent technical challenges in memory management, dependency handling, and tokenizer configuration. On GitHub, bug reports and core functionality issues are the most frequently reported problems, while code review, similarity search, and prompt template design are the most time-consuming to resolve. By uncovering developer practices and pain points, this study points to opportunities to improve FMware tools, workflows, and community support, and provides actionable insights to help guide the future of FMware development.