Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Usage, Effects and Requirements for AI Coding Assistants in the Enterprise: An Empirical Study

Maja Vukovic, Rangeet Pan, Tin Kam Ho, Rahul Krishna, Raju Pavuluri, Michele Merler

TL;DR

The paper investigates whether AI coding assistants powered by CodeLLMs are ready for real-world enterprise use and how they influence software development processes. It combines an empirical study of 57 industry developers with a synthesis of 35 practitioner surveys to characterize usage, productivity gains, and evolving requirements, highlighting substantial perceived benefits and diverse division-specific needs. Key findings show widespread productivity improvements (many reporting 25%+ gains), but also concerns about correctness, security, integration, and governance, along with a lack of long-term organizational outcome analysis. The work underscores a need for customized, repository-aware, and eventually agentic AI coding tools tailored to enterprise contexts, as well as governance-focused research to guide safe and effective deployment.

Abstract

The rise of large language models (LLMs) has accelerated the development of automated techniques and tools for supporting various software engineering tasks, e.g., program understanding, code generation, software testing, and program repair. As CodeLLMs are being employed toward automating these tasks, one question that arises, especially in enterprise settings, is whether these coding assistants and the code LLMs that power them are ready for real-world projects and enterprise use cases, and how do they impact the existing software engineering process and user experience. In this paper we survey 57 developers from different domains and with varying software engineering skill about their experience with AI coding assistants and CodeLLMs. We also reviewed 35 user surveys on the usage, experience and expectations of professionals and students using AI coding assistants and CodeLLMs. Based on our study findings and analysis of existing surveys, we discuss the requirements for AI-powered coding assistants.

Usage, Effects and Requirements for AI Coding Assistants in the Enterprise: An Empirical Study

TL;DR

The paper investigates whether AI coding assistants powered by CodeLLMs are ready for real-world enterprise use and how they influence software development processes. It combines an empirical study of 57 industry developers with a synthesis of 35 practitioner surveys to characterize usage, productivity gains, and evolving requirements, highlighting substantial perceived benefits and diverse division-specific needs. Key findings show widespread productivity improvements (many reporting 25%+ gains), but also concerns about correctness, security, integration, and governance, along with a lack of long-term organizational outcome analysis. The work underscores a need for customized, repository-aware, and eventually agentic AI coding tools tailored to enterprise contexts, as well as governance-focused research to guide safe and effective deployment.

Abstract

The rise of large language models (LLMs) has accelerated the development of automated techniques and tools for supporting various software engineering tasks, e.g., program understanding, code generation, software testing, and program repair. As CodeLLMs are being employed toward automating these tasks, one question that arises, especially in enterprise settings, is whether these coding assistants and the code LLMs that power them are ready for real-world projects and enterprise use cases, and how do they impact the existing software engineering process and user experience. In this paper we survey 57 developers from different domains and with varying software engineering skill about their experience with AI coding assistants and CodeLLMs. We also reviewed 35 user surveys on the usage, experience and expectations of professionals and students using AI coding assistants and CodeLLMs. Based on our study findings and analysis of existing surveys, we discuss the requirements for AI-powered coding assistants.
Paper Structure (23 sections, 12 figures, 2 tables)

This paper contains 23 sections, 12 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Survey on AI coding assistants use in enterprise and academia
  4. Search Selection Criteria
  5. Methodology
  6. Analysis, Gaps and Improvement Areas
  7. Survey
  8. Questionnaire
  9. Participant Demographics
  10. Research Questions
  11. Motivation to use AI coding assistant
  12. Differences across divisions in motivation, use and desired features
  13. Productivity Gains and Code Retention
  14. Help received from the AI coding assistant
  15. Change in software engineering process due to using an AI coding assistant
  16. ...and 8 more sections

Figures (12)

  • Figure 1: Surveys search, selection and analysis process.
  • Figure 2: Distribution of AI Tools (a), Tasks (b) Users Professions (c) and Goals (d) covered in the 35 SE surveys analyzed.
  • Figure 3: Overview of user demographics and technical background.
  • Figure 4: Breakdown programming environments and IDEs the user utilize in their daily development (users were allowed to list up to three).
  • Figure 5: Distribution of AI coding assistants with which the users have had hands on experience in the past (users were allowed to list up to three).
  • ...and 7 more figures