Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisions

TL;DR

The paper introduces Syntactic Framing Fragility (SFF), a robustness audit that isolates purely syntactic polarity effects in LLM ethical decisions by using Logical Polarity Normalization (LPN). It conducts a large-scale cross-origin audit of 23 models across 14 ethical scenarios with four framing variants, measuring fragility with the Syntactic Variation Index (SVI). Key findings show widespread fragility, a pronounced open-source gap, and strong negation sensitivity, with reasoning elicitation offering conditional mitigation. The work argues that syntactic invariance is a critical robustness dimension and recommends SFF-style audits as a standard practice in safety evaluations for high-stakes deployment. These insights have practical implications for deployment risk assessment, prompting scenario-aware reporting and configuration-specific robustness validation, including reasoning-enabled variants.

Abstract

Large language models (LLMs) are increasingly deployed in consequential decision-making settings, yet their robustness to benign prompt variation remains underexplored. In this work, we study whether LLMs maintain consistent ethical judgments across logically equivalent but syntactically different prompts, focusing on variations involving negation and conditional structure. We introduce Syntactic Framing Fragility (SFF), a robustness evaluation framework that isolates purely syntactic effects via Logical Polarity Normalization (LPN), enabling direct comparison of decisions across positive and negative framings without semantic drift. Auditing 23 state-of-the-art models spanning the U.S. and China as well as small U.S. open-source software models over 14 ethical scenarios and four controlled framings (39,975 decisions), we find widespread and statistically significant inconsistency: many models reverse ethical endorsements solely due to syntactic polarity, with open-source models exhibiting over twice the fragility of commercial counterparts. We further uncover extreme negation sensitivity, where some models endorse actions in 80-97% of cases when explicitly prompted with "should not." We show that eliciting chain-of-thought reasoning substantially reduces fragility, identifying a practical mitigation lever, and we map fragility across scenarios, finding higher risk in financial and business contexts than in medical scenarios. Our results demonstrate that syntactic consistency constitutes a distinct and critical dimension of ethical robustness, and we argue that SFF-style audits should be a standard component of safety evaluation for deployed LLMs. Code and results will be available on github.com.

Figures (15)

• Figure 1: Scenario--model fragility heatmap. Cell values show SVI (max--min LPN-normalized action endorsement across four syntactic frames) for each model--scenario pair. Rows are grouped by model origin and columns by scenario, highlighting broad syntactic fragility with concentrated high-risk regions (e.g., financial/business) and comparatively robust scenarios (e.g., medical).
• Figure 2: Model-level syntactic framing fragility. Lollipop ranking of all 23 models by mean SVI (LPN-normalized max--min action endorsement gap across four syntactic frames), colored by origin (US=blue, CN=red, OSS=green). Open-source models cluster at high fragility, while commercial models are more interleaved and include the most robust systems.
• Figure 3: Action endorsement by syntactic frame and origin. Bars show LPN-normalized action endorsement rates across four frames. OSS models exhibit extreme endorsement under negation-bearing frames (F1, F3), indicating a dominant polarity-related failure mode.
• Figure 4: Scenario-level fragility with 95% bootstrap confidence intervals. The patient confidentiality scenario exhibits near-zero fragility and is statistically separated from other ethical dilemmas.
• Figure 5: Effect of reasoning elicitation on fragility (absolute). Paired bars compare SVI for reasoning-enabled and non-reasoning variants, illustrating that deliberative prompting often reduces fragility but with model-dependent magnitude.