The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation

TL;DR

This work uncovers a novel LLM provider bias in code generation, showing that state-of-the-art models preferentially select certain service providers (notably Google and Amazon) and can autonomously modify user code to incorporate favored providers. The authors construct a large, automated pipeline generating 17,014 prompts across 30 real-world scenarios and 6 coding tasks, evaluating seven LLMs and collecting 591,083 valid responses. They introduce two metrics, Gini Index and Modification Ratio, to quantify bias in generation and modification tasks, revealing substantial disparities across models and scenarios, with limited effectiveness from common debiasing prompts. The study highlights significant implications for market dynamics, user autonomy, and security, and provides a public dataset and labeling pipeline to foster reproducibility and future fairness research. Overall, the work emphasizes the need for systematic evaluation and mitigation of provider bias in AI-powered code generation tools to preserve fairness and trust in AI-assisted software development.

Abstract

Large Language Models (LLMs) have emerged as the new recommendation engines, surpassing traditional methods in both capability and scope, particularly in code generation. In this paper, we reveal a novel provider bias in LLMs: without explicit directives, these models show systematic preferences for services from specific providers in their recommendations (e.g., favoring Google Cloud over Microsoft Azure). To systematically investigate this bias, we develop an automated pipeline to construct the dataset, incorporating 6 distinct coding task categories and 30 real-world application scenarios. Leveraging this dataset, we conduct the first comprehensive empirical study of provider bias in LLM code generation across seven state-of-the-art LLMs, utilizing approximately 500 million tokens (equivalent to $5,000+ in computational costs). Our findings reveal that LLMs exhibit significant provider preferences, predominantly favoring services from Google and Amazon, and can autonomously modify input code to incorporate their preferred providers without users' requests. Such a bias holds far-reaching implications for market dynamics and societal equilibrium, potentially contributing to digital monopolies. It may also deceive users and violate their expectations, leading to various consequences. We call on the academic community to recognize this emerging issue and develop effective evaluation and mitigation methods to uphold AI security and fairness.

Figures (13)

• Figure 1: Our study on LLM provider bias is motivated by a real-world case encountered by one of our authors. (a) When the author queries LLMs to debug code snippets that miss variables, (b) the Gemini-1.5-Flash model, developed by Google, completely modifies the code and replaces the intended DragonFly service with the Google Speech Recognition, which is a paid service and not financially supported by our organizations. This increases the development and maintenance costs, which is contrary to the author's intent to utilize a cost-effective, open-source solution. This preference for one's own services may promote monopoly and even lead to legal consequences. (c) In contrast, GPT-3.5-Turbo accurately identifies and fixes the bug when querying with the same inputs. (Green highlights the code snippets modified and added by LLMs)
• Figure 2: The distribution of Gini Index in various scenarios across different models. (Red and yellow separately mark the median and mean GI values for each LLM)
• Figure 3: The preferred providers of LLMs in 'generation' task across 15 scenarios. (Google and Amazon are preferred by LLMs in the most scenarios)
• Figure 4: The distribution of modification cases on different LLMs. (The legend fisplays the abbreviations of coding task)
• Figure 5: The distribution of preferred providers on modification cases across 15 scenarios. (Purple indicates scenarios where LLMs exhibit no modification cases.)