Mining Type Constructs Using Patterns in AI-Generated Code
Imgyeong Lee, Tayyib Ul Hassan, Abram Hindle
TL;DR
The paper investigates how AI agents navigate TypeScript's type system and whether they overuse risky constructs like the any type. It employs a two-stage dataset filtering pipeline combining regex-based prefiltering with a two-agent LLM validation to compare AI-generated PRs with human PRs from the AIDev dataset. It finds that AI agents are substantially more likely to introduce the any type and other advanced type constructs, yet their PRs achieve higher acceptance rates, suggesting potential long-term maintenance risks. The work highlights the need for stricter type-focused review when integrating AI into development workflows to mitigate latent technical debt.
Abstract
Artificial Intelligence (AI) increasingly automates various parts of the software development tasks. Although AI has enhanced the productivity of development tasks, it remains unstudied whether AI essentially outperforms humans in type-related programming tasks, such as employing type constructs properly for type safety, during its tasks. Moreover, there is no systematic study that evaluates whether AI agents overuse or misuse the type constructs under the complicated type systems to the same extent as humans. In this study, we present the first empirical analysis to answer these questions in the domain of TypeScript projects. Our findings show that, in contrast to humans, AI agents are 9x more prone to use the 'any' keyword. In addition, we observed that AI agents use advanced type constructs, including those that ignore type checks, more often compared to humans. Surprisingly, even with all these issues, Agentic pull requests (PRs) have 1.8x higher acceptance rates compared to humans for TypeScript. We encourage software developers to carefully confirm the type safety of their codebases whenever they coordinate with AI agents in the development process.