Redefining Developer Assistance: Through Large Language Models in Software Ecosystem
Somnath Banerjee, Avik Dutta, Sayan Layek, Amruit Sahoo, Sam Conrad Joyce, Rima Hazra
TL;DR
The paper addresses the need for software-domain assistance beyond coding by introducing DevAssistLlama, an instruction-tuned LLM based on Llama 2-13B optimized for software-related tasks. It constructs a large, multi-task instruction dataset from diverse software sources and applies LoRA-based fine-tuning to align the model with NER, RE, LP, FAR, and QA demands, while employing negative sampling to reduce irrelevant outputs. Empirical results show DevAssistLlama achieving state-of-the-art macro F1 on NER/RE/LP and strong FAR performance, though ChatGPT can outperform it on QA tasks, highlighting trade-offs across domains. The work demonstrates the practical potential of domain-specific LLMs to augment software developer workflows, with implications for tooling in documentation understanding, code-related tasks, and community Q&A. Limitations identified include terminology depth, code generation quality, data biases, and adaptation speed, with future work focusing on richer software datasets, closer developer collaboration, and mechanisms for real-time learning.
Abstract
In this paper, we delve into the advancement of domain-specific Large Language Models (LLMs) with a focus on their application in software development. We introduce DevAssistLlama, a model developed through instruction tuning, to assist developers in processing software-related natural language queries. This model, a variant of instruction tuned LLM, is particularly adept at handling intricate technical documentation, enhancing developer capability in software specific tasks. The creation of DevAssistLlama involved constructing an extensive instruction dataset from various software systems, enabling effective handling of Named Entity Recognition (NER), Relation Extraction (RE), and Link Prediction (LP). Our results demonstrate DevAssistLlama's superior capabilities in these tasks, in comparison with other models including ChatGPT. This research not only highlights the potential of specialized LLMs in software development also the pioneer LLM for this domain.