Advancing Automated Ethical Profiling in SE: a Zero-Shot Evaluation of LLM Reasoning

TL;DR

This work addresses the challenge of integrating ethical reasoning into software engineering by evaluating 16 pre-trained LLMs in a strict zero-shot setting across 30 ethically charged scenarios. It introduces a fully automated framework that prompts models to select an ethical theory, judge action acceptability, and produce explanations, then compares results to three expert ethicists using Theory Consistency Rate ($TCR$) and Binary Agreement Rate ($BAR$), complemented by qualitative analyses of explanations. The findings show substantial inter-model agreement on acceptability ($BAR\approx86.7\%$) and moderate agreement on theory ($TCR\approx73.3\%$), with explanations that are semantically coherent but lexically diverse. The study demonstrates the viability of using LLMs as modular ethical reasoning components in SE pipelines, while recommending ensemble approaches and human oversight for ethically ambiguous cases, and provides a reproducible benchmark for auditing moral inference in engineering contexts.

Abstract

Large Language Models (LLMs) are increasingly integrated into software engineering (SE) tools for tasks that extend beyond code synthesis, including judgment under uncertainty and reasoning in ethically significant contexts. We present a fully automated framework for assessing ethical reasoning capabilities across 16 LLMs in a zero-shot setting, using 30 real-world ethically charged scenarios. Each model is prompted to identify the most applicable ethical theory to an action, assess its moral acceptability, and explain the reasoning behind their choice. Responses are compared against expert ethicists' choices using inter-model agreement metrics. Our results show that LLMs achieve an average Theory Consistency Rate (TCR) of 73.3% and Binary Agreement Rate (BAR) on moral acceptability of 86.7%, with interpretable divergences concentrated in ethically ambiguous cases. A qualitative analysis of free-text explanations reveals strong conceptual convergence across models despite surface-level lexical diversity. These findings support the potential viability of LLMs as ethical inference engines within SE pipelines, enabling scalable, auditable, and adaptive integration of user-aligned ethical reasoning. Our focus is the Ethical Interpreter component of a broader profiling pipeline: we evaluate whether current LLMs exhibit sufficient interpretive stability and theory-consistent reasoning to support automated profiling.

Figures (12)

• Figure 1: Automated Ethical Profile Generation
• Figure 2: Approach overview of this research
• Figure 3: LLMs TCR and BAR results with Fleiss' Kappa agreement coloring and Z-Scores with threshold coloring.
• Figure 4: PCA Clustering
• Figure 5: t-SNE Clustering