Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM Agents Improve Semantic Code Search

Sarthak Jain, Aditya Dora, Ka Seng Sam, Prabhat Singh

TL;DR

The paper tackles semantic code search under ambiguity and vocabulary mismatch by leveraging agentic LLMs with Retrieval Augmented Generation to inject repository-specific context into queries. It introduces a multi-stream ensemble that translates natural language queries into code embeddings and cross-compares across multiple streams to improve retrieval accuracy. The RepoRift platform implements these techniques and demonstrates state-of-the-art performance on CodeSearchNet's Python data, achieving 78.2% at Success@10 and 34.6% at Success@1. This work offers a practical, context-aware approach to code search that blends prompt augmentation with ensemble retrieval to enhance relevance and precision in real-world developer workflows.

Abstract

Code Search is a key task that many programmers often have to perform while developing solutions to problems. Current methodologies suffer from an inability to perform accurately on prompts that contain some ambiguity or ones that require additional context relative to a code-base. We introduce the approach of using Retrieval Augmented Generation (RAG) powered agents to inject information into user prompts allowing for better inputs into embedding models. By utilizing RAG, agents enhance user queries with relevant details from GitHub repositories, making them more informative and contextually aligned. Additionally, we introduce a multi-stream ensemble approach which when paired with agentic workflow can obtain improved retrieval accuracy, which we deploy on application called repo-rift.com. Experimental results on the CodeSearchNet dataset demonstrate that RepoRift significantly outperforms existing methods, achieving an 78.2% success rate at Success@10 and a 34.6% success rate at Success@1. This research presents a substantial advancement in semantic code search, highlighting the potential of agentic LLMs and RAG to enhance code retrieval systems.

LLM Agents Improve Semantic Code Search

TL;DR

The paper tackles semantic code search under ambiguity and vocabulary mismatch by leveraging agentic LLMs with Retrieval Augmented Generation to inject repository-specific context into queries. It introduces a multi-stream ensemble that translates natural language queries into code embeddings and cross-compares across multiple streams to improve retrieval accuracy. The RepoRift platform implements these techniques and demonstrates state-of-the-art performance on CodeSearchNet's Python data, achieving 78.2% at Success@10 and 34.6% at Success@1. This work offers a practical, context-aware approach to code search that blends prompt augmentation with ensemble retrieval to enhance relevance and precision in real-world developer workflows.

Abstract

Code Search is a key task that many programmers often have to perform while developing solutions to problems. Current methodologies suffer from an inability to perform accurately on prompts that contain some ambiguity or ones that require additional context relative to a code-base. We introduce the approach of using Retrieval Augmented Generation (RAG) powered agents to inject information into user prompts allowing for better inputs into embedding models. By utilizing RAG, agents enhance user queries with relevant details from GitHub repositories, making them more informative and contextually aligned. Additionally, we introduce a multi-stream ensemble approach which when paired with agentic workflow can obtain improved retrieval accuracy, which we deploy on application called repo-rift.com. Experimental results on the CodeSearchNet dataset demonstrate that RepoRift significantly outperforms existing methods, achieving an 78.2% success rate at Success@10 and a 34.6% success rate at Success@1. This research presents a substantial advancement in semantic code search, highlighting the potential of agentic LLMs and RAG to enhance code retrieval systems.
Paper Structure (12 sections, 20 equations, 1 figure, 1 table)

This paper contains 12 sections, 20 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Information Injection via Agentic Large Language Models (LLMs) and Retrieval Augmented Generation (RAG)
  4. Ensemble Architecture with Multi-Stream Comparisons
  5. Experimental Setup
  6. Dataset
  7. Implementation Details
  8. Evaluation Metrics
  9. Results
  10. Conclusion
  11. Future Work
  12. Appendix

Figures (1)

  • Figure 1: An example showing the idea of how a natural langauge query taken fron CodeSearchNet husain2020codesearchnetchallengeevaluatingstate is augmented by Agents allowing for better matching.