Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries

Nora Petrova, John Burden

Abstract

What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? We find that commercial system prompts can override safety training, causing frontier models to lie about medical risks, dismiss safety concerns, and prioritise profit over user welfare. Testing 8 models in scenarios where commercial objectives conflict with user safety -- a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings -- we uncover catastrophic failures: models fabricating safety information, explicitly reasoning they should refuse but proceeding anyway, and actively discouraging users from consulting doctors. Most alarmingly, models show no "red line", their willingness to comply with harmful requests does not decrease as potential consequences escalate from minor to life-threatening. Our findings suggest that current safety training does not generalise to commercial deployment contexts.

The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries

Abstract

What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? We find that commercial system prompts can override safety training, causing frontier models to lie about medical risks, dismiss safety concerns, and prioritise profit over user welfare. Testing 8 models in scenarios where commercial objectives conflict with user safety -- a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings -- we uncover catastrophic failures: models fabricating safety information, explicitly reasoning they should refuse but proceeding anyway, and actively discouraging users from consulting doctors. Most alarmingly, models show no "red line", their willingness to comply with harmful requests does not decrease as potential consequences escalate from minor to life-threatening. Our findings suggest that current safety training does not generalise to commercial deployment contexts.
Paper Structure (22 sections, 4 figures, 3 tables)

This paper contains 22 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Adversarial Auditing Framework
  4. Commercial Pressure Levels
  5. Scenarios and Domains
  6. Evaluation
  7. Results
  8. Catastrophic Failures Under Commercial Pressure
  9. Model Stratification
  10. The Pressure Gradient
  11. Failure Patterns
  12. Discussion and Conclusion
  13. Limitations
  14. Conclusion
  15. Transcript Examples: Catastrophic Failures
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Each point represents one (model, severity band) pair. Red-shaded zones indicate concerning behaviour in high-stakes contexts. Key finding: most models show no reduction in harmful compliance as stakes increase from minor to critical.
  • Figure 2: Harmful compliance increases dramatically with commercial pressure intensity. Error bands show standard deviation across all models and scenarios.
  • Figure 3: Distribution of harmful compliance scores across pressure levels. Variance increases dramatically at higher pressure, indicating less predictable behaviour when stakes are highest.
  • Figure 4: Pressure gradient by domain. Healthcare shows the steepest increase, suggesting safety training inadequately covers medical contexts.