Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Scientists Use Large Language Models to Program

Gabrielle O'Brien

TL;DR

The paper investigates how scientists who code engage with code-generating large language models (Code LLMs) and which interfaces—browser-based Chat vs IDE-embedded Copilot—they favor. Through a university-scale survey (n=199) and 14 in-depth interviews complemented by interaction logs, the authors find that Code LLMs function predominantly as information-retrieval aids for learning new languages and libraries, with verification largely relying on running code, eyeballing outputs, and reading generated code. The study highlights suboptimal verification practices and prevalent misconceptions about how Code LLMs work, underscoring the need for better tooling and interfaces that distinguish retrieval from generation, improve error signaling, and support rigorous code validation in scientific contexts. Limitations include self-reported data and a single-resource university sample, suggesting replication across diverse institutions to generalize the findings and inform design of robust, discipline-aware code-verification tools.

Abstract

Scientists across disciplines write code for critical activities like data collection and generation, statistical modeling, and visualization. As large language models that can generate code have become widely available, scientists may increasingly use these models during research software development. We investigate the characteristics of scientists who are early-adopters of code generating models and conduct interviews with scientists at a public, research-focused university. Through interviews and reviews of user interaction logs, we see that scientists often use code generating models as an information retrieval tool for navigating unfamiliar programming languages and libraries. We present findings about their verification strategies and discuss potential vulnerabilities that may emerge from code generation practices unknowingly influencing the parameters of scientific analyses.

How Scientists Use Large Language Models to Program

TL;DR

The paper investigates how scientists who code engage with code-generating large language models (Code LLMs) and which interfaces—browser-based Chat vs IDE-embedded Copilot—they favor. Through a university-scale survey (n=199) and 14 in-depth interviews complemented by interaction logs, the authors find that Code LLMs function predominantly as information-retrieval aids for learning new languages and libraries, with verification largely relying on running code, eyeballing outputs, and reading generated code. The study highlights suboptimal verification practices and prevalent misconceptions about how Code LLMs work, underscoring the need for better tooling and interfaces that distinguish retrieval from generation, improve error signaling, and support rigorous code validation in scientific contexts. Limitations include self-reported data and a single-resource university sample, suggesting replication across diverse institutions to generalize the findings and inform design of robust, discipline-aware code-verification tools.

Abstract

Scientists across disciplines write code for critical activities like data collection and generation, statistical modeling, and visualization. As large language models that can generate code have become widely available, scientists may increasingly use these models during research software development. We investigate the characteristics of scientists who are early-adopters of code generating models and conduct interviews with scientists at a public, research-focused university. Through interviews and reviews of user interaction logs, we see that scientists often use code generating models as an information retrieval tool for navigating unfamiliar programming languages and libraries. We present findings about their verification strategies and discuss potential vulnerabilities that may emerge from code generation practices unknowingly influencing the parameters of scientific analyses.

Paper Structure

This paper contains 34 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Background and Related work
  4. How programmers use Code LLMs
  5. How data analysts use Code LLMs
  6. How novice programmers use Code LLMs
  7. A survey of scientists who use Code LLMs
  8. Methods
  9. Preparing survey responses for analysis
  10. Results
  11. Programming experience
  12. Code LLM usage
  13. Predictors of Code LLM usage frequency
  14. Relationships with research field
  15. Interim summary of findings
  16. ...and 19 more sections

Figures (1)

  • Figure 1: Correlation matrix for programming-related survey variables. Stars indicate statistical significance at the 0.05 (*), 0.01 (**), and 0.001 (***) levels. All $p$-values are shown uncorrected. Black rectangles indicate clusters of predictors determined by hierarchical clustering with 5 clusters.