When Is It Acceptable to Break the Rules? Knowledge Representation of Moral Judgement Based on Empirical Data

TL;DR

This paper investigates how humans judge the acceptability of breaking a simple norm (no cutting in line) across realistic contexts, revealing that moral judgments are graded and influenced by context, welfare, and universalization considerations. It advances a computational framework, SEP-nets, a three-layer generalization of CP-nets that incorporates Scenario, Evaluation, and Preference variables to capture moral reasoning in AI. The results demonstrate that both outcome-based and agreement-based (contractualist) System 2 processes shape judgments, while System 1 rules can also be decisive in some cases, supporting a Psychological Triple Theory of morality. The work lays a foundation for embedding morally flexible, context-aware judgments into AI systems and outlines future directions for learning, evaluation, and prescriptive planning based on moral preferences.

Abstract

One of the most remarkable things about the human moral mind is its flexibility. We can make moral judgments about cases we have never seen before. We can decide that pre-established rules should be broken. We can invent novel rules on the fly. Capturing this flexibility is one of the central challenges in developing AI systems that can interpret and produce human-like moral judgment. This paper details the results of a study of real-world decision makers who judge whether it is acceptable to break a well-established norm: ``no cutting in line.'' We gather data on how human participants judge the acceptability of line-cutting in a range of scenarios. Then, in order to effectively embed these reasoning capabilities into a machine, we propose a method for modeling them using a preference-based structure, which captures a novel modification to standard ``dual process'' theories of moral judgment.

Figures (7)

• Figure 1: Moral judgements about cutting in line in each of the scenarios. Color indicates if the person cutting in line is requesting the main service or not (blue for Yes, red for No). Error bars are $95\%$ confidence intervals. As we can see, a simple rule, such as "it is ok to cut if you are not requesting the main service" is not sufficient to explain variation.
• Figure 2: Box-plots depicting the distribution of responses to the evaluation questions, polled across scenarios.
• Figure 3: Cross-correlation matrix for all scenarios. The moral judgement is labeled as PREF and is negatively correlated with all the evaluating metrics, indicating that as any measure gets worse, the moral permissibility goes down.
• Figure 4: The CP-net representing John's preferences described in Example \ref{['example1']}. Variable $S$ denotes the scenario: $a$ for "at the airport", and $\overline{a}$ for "not at the airport"; variable $T$ represents time: $o$ for "on time", and $\overline{o}$ for "not on time"; and a preference variable $P$ represent the judgement over cutting the line: i.e., $c$ is for "ok to cut the line", and $\overline{c}$ is for "not ok to cut the line".
• Figure 5: The preorder induced by the CP-net in Figure \ref{['excpnet1']}. The two components are due to the indifference on the domain of the variable $A$.

Theorems & Definitions (2)

• Example 1
• Definition 1