Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant

TL;DR

This study investigates the challenges of building contextualized LLM-based coding assistants through StackSpot AI CodeBuddy. By combining interviews with developers and analysis of issue logs, the authors extract 13 practical lessons spanning LLM behavior, user interaction, and technical architecture, with a particular emphasis on retrieval-augmented generation to inject organization-specific knowledge. The work demonstrates how RAG, token-management, and careful UX considerations can yield more accurate, transparent, and useful coding assistance, while highlighting the ongoing need for experimentation and cross-disciplinary collaboration. The findings offer actionable guidance for teams aiming to deploy AI-powered coding assistants in real-world development environments and underscore the practical impact of contextualization on software productivity and trust in AI tools.

Abstract

With their exceptional natural language processing capabilities, tools based on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become indispensable resources in the software developer's toolkit. While recent studies suggest the potential productivity gains these tools can unlock, users still encounter drawbacks, such as generic or incorrect answers. Additionally, the pursuit of improved responses often leads to extensive prompt engineering efforts, diverting valuable time from writing code that delivers actual value. To address these challenges, a new breed of tools, built atop LLMs, is emerging. These tools aim to mitigate drawbacks by employing techniques like fine-tuning or enriching user prompts with contextualized information. In this paper, we delve into the lessons learned by a software development team venturing into the creation of such a contextualized LLM-based application, using retrieval-based techniques, called CodeBuddy. Over a four-month period, the team, despite lacking prior professional experience in LLM-based applications, built the product from scratch. Following the initial product release, we engaged with the development team responsible for the code generative components. Through interviews and analysis of the application's issue tracker, we uncover various intriguing challenges that teams working on LLM-based applications might encounter. For instance, we found three main group of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By understanding these lessons, software development teams could become better prepared to build LLM-based applications.

Figures (1)

• Figure 1: StackSpot AI's view in VS Code.