A Multi-agent Onboarding Assistant based on Large Language Models, Retrieval Augmented Generation, and Chain-of-Thought
Andrei Cristian Ionescu, Sergey Titov, Maliheh Izadi
TL;DR
The paper tackles the challenge of onboarding in rapidly evolving software projects by introducing Onboarding Buddy, an agent-based system that combines large language models, retrieval-augmented generation, and automated chain-of-thought to deliver dynamic, project-specific onboarding within the development environment. It features an Onboarding Agent core with memory, planning scratchpad, and retrieval tools, using a FAISS-based semantic search and a dual-tool retrieval strategy to ground reasoning in the project codebase. An empirical evaluation with eight participants demonstrated favorable perceived helpfulness and ease of onboarding, with high task completion and constructive UX feedback, while acknowledging scalability and generalizability limitations. The work suggests Onboarding Buddy can reduce mentor burden, accelerate ramp-up, and improve developer satisfaction, with future plans for robustness, broader benchmarking, and live-codebase updates.
Abstract
Effective onboarding in software engineering is crucial but difficult due to the fast-paced evolution of technologies. Traditional methods, like exploration and workshops, are costly, time-consuming, and quickly outdated in large projects. We propose the Onboarding Buddy system, which leverages large language models, retrieval augmented generation, and an automated chain-of-thought approach to improve onboarding. It integrates dynamic, context-specific support within the development environment, offering natural language explanations, code insights, and project guidance. Our solution is agent-based and provides customized assistance with minimal human intervention. Our study results among the eight participants show an average helpfulness rating of (M=3.26, SD=0.86) and ease of onboarding at (M=3.0, SD=0.96) out of four. While similar to tools like GitHub Copilot, Onboarding Buddy uniquely integrates a chain-of-thought reasoning mechanism with retrieval-augmented generation, tailored specifically for dynamic onboarding contexts. While our initial evaluation is based on eight participants within one project, we will explore larger teams and multiple real-world codebases in the company to demonstrate broader applicability. Overall, Onboarding Buddy holds great potential for enhancing developer productivity and satisfaction. Our tool, source code, and demonstration video are publicly available