Requirements for Aligned, Dynamic Resolution of Conflicts in Operational Constraints

TL;DR

Autonomous agents operating in open environments face numerous, often conflicting constraints that cannot be fully addressed by pre-deployment training alone. The paper proposes a knowledge-level framework called Online Aligned Mitigation of Novel Constraint Conflicts (OAMNCC) and a taxonomy of knowledge types to support online, justified decision-making when novel conflicts arise. Through scenario-driven analysis (e.g., Sailor Overboard, Piracy Interdiction) and a process model mapping knowledge types to decision steps, it shows how frames, expressive preferences, action affordances, dynamic situation models, information quality, and metaknowledge enable grounded, human-aligned action, including reframing constraints as needed. This work lays a foundation for designing evaluable, robust decision systems capable of adapting to ill-structured, real-world environments while remaining aligned with human values.

Abstract

Deployed, autonomous AI systems must often evaluate multiple plausible courses of action (extended sequences of behavior) in novel or under-specified contexts. Despite extensive training, these systems will inevitably encounter scenarios where no available course of action fully satisfies all operational constraints (e.g., operating procedures, rules, laws, norms, and goals). To achieve goals in accordance with human expectations and values, agents must go beyond their trained policies and instead construct, evaluate, and justify candidate courses of action. These processes require contextual "knowledge" that may lie outside prior (policy) training. This paper characterizes requirements for agent decision making in these contexts. It also identifies the types of knowledge agents require to make decisions robust to agent goals and aligned with human expectations. Drawing on both analysis and empirical case studies, we examine how agents need to integrate normative, pragmatic, and situational understanding to select and then to pursue more aligned courses of action in complex, real-world environments.

Figures (9)

• Figure 1: Illustration of Sailor Overboard Scenario.
• Figure 2: Utilitarian assessment of sailor overboard scenario. Colored dots represent different safety margins. Identical marks vary in the ratio of rescue/RTB importance. $\times$ represents a policy that never leaves a sailor behind once spotted.
• Figure 3: Illustration of the Piracy Interdiction Scenarios.
• Figure 4: Distributions of the ransom avoided by interdiction relative to baseline (no action) for different decision strategies (1000 trials; purple, horizontal lines identify means). The distributions are significantly different according to Kolmogorov-Smirnov tests.
• Figure 5: Distributions of ransom avoided for each strategy when two merchant vessels can deploy water cannons. Left of dashed vertical line: the agent ignores water cannon capabilities.