FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion

TL;DR

FT2Ra introduces a theoretically grounded retrieval-augmented approach for code completion that emulates fine-tuning by using an iterative retrieval loop and delta-logits. By deriving a practical approximation for the change in logits, FT2Ra stores and leverages fine-grained retrieval information in a datastore and updates predictions without full parameter updates. Empirical results on token-level and line-level tasks show FT2Ra outperforms strong baselines and can approach, or surpass, fine-tuned models in several settings, with performance improving further through additional iterations. The work provides a principled alternative to fine-tuning for large code models and demonstrates substantial gains across multiple benchmarks and architectures, with open-source resources for reproducibility.

Abstract

The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions. To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning.In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.

Figures (5)

• Figure 1: Case study for CodeGPT line-level completion on PY150
• Figure 2: Venn diagram of the EM results on CodeGPT (left) and UniXcoder (right).
• Figure 3: Results of token-level completion on fine-tuned models with different epochs.
• Figure 4: Results of line-level completion.
• Figure 5: Results with different numbers of iterations.