Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DepsRAG: Towards Agentic Reasoning and Planning for Software Dependency Management

Mohannad Alhanahnah, Yazan Boshmaf

TL;DR

This paper introduces DepsRAG, a multi-agent framework designed to assist developers in reasoning about software dependencies, and assesses its adaptability to novel scenarios using GPT-4-Turbo and Llama-3 on three multi-step reasoning tasks.

Abstract

In the era of Large Language Models (LLMs) with their advanced capabilities, a unique opportunity arises to develop LLM-based digital assistant tools that can support software developers by facilitating comprehensive reasoning about software dependencies and open-source libraries before importing them. This reasoning process is daunting, mandating multiple specialized tools and dedicated expertise, each focusing on distinct aspects (e.g., security analysis tools may overlook design flaws such as circular dependencies, which hinder software maintainability). Creating a significant bottleneck in the software development lifecycle. In this paper, we introduce DepsRAG, a multi-agent framework designed to assist developers in reasoning about software dependencies. DepsRAG first constructs a comprehensive Knowledge Graph (KG) that includes both direct and transitive dependencies. Developers can interact with DepsRAG through a conversational interface, posing queries about the dependencies. DepsRAG employs Retrieval-Augmented Generation (RAG) to enhance these queries by retrieving relevant information from the KG as well as external sources, such as the Web and vulnerability databases, thus demonstrating its adaptability to novel scenarios. DepsRAG incorporates a Critic-Agent feedback loop to ensure the accuracy and clarity of LLM-generated responses. We evaluated DepsRAG using GPT-4-Turbo and Llama-3 on three multi-step reasoning tasks, observing a threefold increase in accuracy with the integration of the Critic-Agent mechanism. DepsRAG demo and implementation are available: https://github.com/Mohannadcse/DepsRAG.

DepsRAG: Towards Agentic Reasoning and Planning for Software Dependency Management

TL;DR

This paper introduces DepsRAG, a multi-agent framework designed to assist developers in reasoning about software dependencies, and assesses its adaptability to novel scenarios using GPT-4-Turbo and Llama-3 on three multi-step reasoning tasks.

Abstract

In the era of Large Language Models (LLMs) with their advanced capabilities, a unique opportunity arises to develop LLM-based digital assistant tools that can support software developers by facilitating comprehensive reasoning about software dependencies and open-source libraries before importing them. This reasoning process is daunting, mandating multiple specialized tools and dedicated expertise, each focusing on distinct aspects (e.g., security analysis tools may overlook design flaws such as circular dependencies, which hinder software maintainability). Creating a significant bottleneck in the software development lifecycle. In this paper, we introduce DepsRAG, a multi-agent framework designed to assist developers in reasoning about software dependencies. DepsRAG first constructs a comprehensive Knowledge Graph (KG) that includes both direct and transitive dependencies. Developers can interact with DepsRAG through a conversational interface, posing queries about the dependencies. DepsRAG employs Retrieval-Augmented Generation (RAG) to enhance these queries by retrieving relevant information from the KG as well as external sources, such as the Web and vulnerability databases, thus demonstrating its adaptability to novel scenarios. DepsRAG incorporates a Critic-Agent feedback loop to ensure the accuracy and clarity of LLM-generated responses. We evaluated DepsRAG using GPT-4-Turbo and Llama-3 on three multi-step reasoning tasks, observing a threefold increase in accuracy with the integration of the Critic-Agent mechanism. DepsRAG demo and implementation are available: https://github.com/Mohannadcse/DepsRAG.
Paper Structure (16 sections, 6 figures, 3 tables)

This paper contains 16 sections, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Dependency Graphs
  4. Knowledge Graphs (KGs)
  5. Agentic LLM Applications
  6. Motivating Example and Challenges
  7. DepsRAG: Multi-Agent System for Dependency Management
  8. DepsRAG Agent-Critic Interaction and Orchestration Architecture
  9. DepsRAG Proof-of-Concept
  10. Implementation
  11. Experiments
  12. RQ1: Accuracy of Cypher Query Generation
  13. RQ2: Efficiency of the Critic-Agent interaction
  14. Related Work
  15. Conclusion and Future Work
  16. ...and 1 more sections

Figures (6)

  • Figure 1: A real-world demonstration of DepsRAG, applied to identify critical packages and associated risks within the dependency graph of the Chainlit package version 1.1.200. This process involves a sequence of subtasks executed by two agents: DependencyGraphAgent and SearchAgent. The question in Sub-Task 3 will be repeated for each package in the LLM response in Sub-Task 2. Therefore, AssistantAgent needs to orchestrate the communication between these agents, by setting the field "target_agent" in JSON message generated by QuestionTool, aggregate the responses received in Sub-Task 2 and Sub-Task 3, and subsequently forward the agents' responses to the CriticAgent for validation (omitted in this Figure for the sake of simplicity).
  • Figure 2: DepsRAG high-level orchestration architecture
  • Figure 3: Ablation results show the correctness of the answers with and without Critic-Agent interaction.
  • Figure 4: Total number of terminatiom, AssistantAgent questions, and CriticAgent responses.
  • Figure 5: Dependency graphs generated by DepsRAG for TensorFlow across 3 ecosystems (PyPI, NPM, and Rust). Versions 2.16.1, 0.7.0, and 0.21.0, respectively.
  • ...and 1 more figures