RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry

TL;DR

Industrial code completion faces a distributional gap between open-source training data and proprietary enterprise code. The authors compare Retrieval-Augmented Generation (RAG) and Fine-Tuning (FT) for adapting Large Code Models to a Tencent WXG C++ codebase, using 160k files and evaluating six LCMs on effectiveness, efficiency, and parameter sensitivity. They find that RAG with BM25 achieves higher accuracy than FT, while combining RAG and FT yields further gains, and RAG scales more gracefully with larger codebases. The study provides practical guidance for practitioners on when to adopt RAG, FT, or a combination, and highlights directions for embedding improvements and long-context optimization. The findings advance understanding of domain adaptation for industrial code completion with direct implications for enterprise deployments.

Abstract

Code completion, a crucial practice in industrial settings, helps developers improve programming efficiency by automatically suggesting code snippets during development. With the emergence of Large Code Models (LCMs), this field has witnessed significant advancements. Due to the natural differences between open-source and industrial codebases, such as coding patterns and unique internal dependencies, it is a common practice for developers to conduct domain adaptation when adopting LCMs in industry. There exist multiple adaptation approaches, among which retrieval-augmented generation (RAG) and fine-tuning are the two most popular paradigms. However, no prior research has explored the trade-off of the two approaches in industrial scenarios. To mitigate the gap, we comprehensively compare the two paradigms including Retrieval-Augmented Generation (RAG) and Fine-tuning (FT), for industrial code completion in this paper. In collaboration with Tencent's WXG department, we collect over 160,000 internal C++ files as our codebase. We then compare the two types of adaptation approaches from three dimensions that are concerned by industrial practitioners, including effectiveness, efficiency, and parameter sensitivity, using six LCMs. Our findings reveal that RAG, when implemented with appropriate embedding models that map code snippets into dense vector representations, can achieve higher accuracy than fine-tuning alone. Specifically, BM25 presents superior retrieval effectiveness and efficiency among studied RAG methods. Moreover, RAG and fine-tuning are orthogonal and their combination leads to further improvement. We also observe that RAG demonstrates better scalability than FT, showing more sustained performance gains with larger scales of codebase.

Figures (4)

• Figure 1: Average exact match and throughput under different retrieved code snippets across six LCMs.
• Figure 2: Exact match of the base and fine-tuned model under different retrieved contexts.
• Figure 3: Exact match and edit similarity of fine-tuning and RAG under different sizes of codebase.
• Figure 4: Example that the QC-7B base model fails to predict the next line of code but fine-tuning and RAG succeed.