Once Upon a Team: Investigating Bias in LLM-Driven Software Team Composition and Task Allocation

TL;DR

The paper investigates whether LLMs reproduce demographic biases in software team composition and task allocation when both country and pronouns are considered. It uses three LLMs and 3,000 simulated decisions based on GitHub profiles across five countries, analyzing selection and task-assignment outcomes with logistic and multinomial regression, including post-hoc corrections. The findings reveal systematic disparities: pronoun usage and nationality influence both who is selected and which tasks they receive, beyond explicit expertise indicators. These results highlight the socio-technical nature of fairness in LLM-driven SE processes and argue for fairness-aware evaluation, audits across multiple attributes, and human oversight to mitigate disparate treatment.

Abstract

LLMs are increasingly used to boost productivity and support software engineering tasks. However, when applied to socially sensitive decisions such as team composition and task allocation, they raise concerns of fairness. Prior studies have revealed that LLMs may reproduce stereotypes; however, these analyses remain exploratory and examine sensitive attributes in isolation. This study investigates whether LLMs exhibit bias in team composition and task assignment by analyzing the combined effects of candidates' country and pronouns. Using three LLMs and 3,000 simulated decisions, we find systematic disparities: demographic attributes significantly shaped both selection likelihood and task allocation, even when accounting for expertise-related factors. Task distributions further reflected stereotypes, with technical and leadership roles unevenly assigned across groups. Our findings indicate that LLMs exacerbate demographic inequities in software engineering contexts, underscoring the need for fairness-aware assessment.