Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agent-Driven Automatic Software Improvement

Fernando Vallecillos Ruiz

TL;DR

The paper addresses the high maintenance cost and reliability issues in software development by proposing a doctoral program to study LLM-based agents for automatic software improvement. It advocates an iterative, multi-agent framework where agents decompose tasks, learn from critiques, and collaboratively refine code, aiming to overcome last-mile errors and model limitations. The research plan comprises three phases: foundational literature review and experimental design, implementation and empirical testing of single- and multi-agent setups, and comprehensive analysis with novel fine-tuning methodologies. The anticipated outcome is a robust, scalable framework that improves code quality, security, and efficiency through continuous, feedback-driven learning and collaboration among specialized agents, with broad implications for automated software engineering practices.

Abstract

With software maintenance accounting for 50% of the cost of developing software, enhancing code quality and reliability has become more critical than ever. In response to this challenge, this doctoral research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs) to perform software maintenance tasks. The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation. One distinct challenge is the last-mile problems, errors at the final stage of producing functionally and contextually relevant code. Furthermore, this project aims to surpass the inherent limitations of current LLMs in source code through a collaborative framework where agents can correct and learn from each other's errors. We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement. Our main goal is to achieve a leap forward in the field of automatic software improvement by developing new tools and frameworks that can enhance the efficiency and reliability of software development.

Agent-Driven Automatic Software Improvement

TL;DR

The paper addresses the high maintenance cost and reliability issues in software development by proposing a doctoral program to study LLM-based agents for automatic software improvement. It advocates an iterative, multi-agent framework where agents decompose tasks, learn from critiques, and collaboratively refine code, aiming to overcome last-mile errors and model limitations. The research plan comprises three phases: foundational literature review and experimental design, implementation and empirical testing of single- and multi-agent setups, and comprehensive analysis with novel fine-tuning methodologies. The anticipated outcome is a robust, scalable framework that improves code quality, security, and efficiency through continuous, feedback-driven learning and collaboration among specialized agents, with broad implications for automated software engineering practices.

Abstract

With software maintenance accounting for 50% of the cost of developing software, enhancing code quality and reliability has become more critical than ever. In response to this challenge, this doctoral research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs) to perform software maintenance tasks. The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation. One distinct challenge is the last-mile problems, errors at the final stage of producing functionally and contextually relevant code. Furthermore, this project aims to surpass the inherent limitations of current LLMs in source code through a collaborative framework where agents can correct and learn from each other's errors. We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement. Our main goal is to achieve a leap forward in the field of automatic software improvement by developing new tools and frameworks that can enhance the efficiency and reliability of software development.
Paper Structure (14 sections, 3 figures)

This paper contains 14 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background and Challenges
  3. Automatic Software Improvement
  4. Large Language Models for Code
  5. Large Language Model-based Agents
  6. Expected Contributions
  7. Research questions and scientific challenges
  8. Research Agenda
  9. Phase 1: Foundational Research and Experimental Design
  10. Phase 2: Implementation and Empirical Testing
  11. Phase 3: Analysis and Methodological Advancements
  12. Current Status
  13. Threats to Validity and Mitigation Strategies
  14. Conclusion

Figures (3)

  • Figure 1: Conceptual framework for an LLM agent.
  • Figure 2: Scenarios of LLM-based agents interactions.
  • Figure 3: Overview of the research phases and the steps inside each phase.