Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InvisibleBench: A Deployment Gate for Caregiving Relationship AI

Ali Madad

TL;DR

InvisibleBench introduces a longitudinal, five-dimension deployment gate for caregiving AI, extending beyond single-turn safety tests by evaluating 3-20+ turn interactions across Safety, Compliance, Trauma, Belonging, and Memory. It combines deterministic autofail criteria with an LLM-assisted judging framework and a three-tier (tiered) multi-turn design to detect subtle, time-evolving harms such as attachment, boundary creep, and regulatory violations. Across 68 evaluations of four frontier models, the study reveals universal gaps in crisis detection and large variance in regulatory compliance, arguing for deterministic crisis routing and hybrid architectures to mitigate deployment risk. The work provides open-scoped artifacts, scalable evaluation costs, and a practical path toward safer, deployment-ready caregiving AI with a transparent, community-extensible methodology.

Abstract

InvisibleBench is a deployment gate for caregiving-relationship AI, evaluating 3-20+ turn interactions across five dimensions: Safety, Compliance, Trauma-Informed Design, Belonging/Cultural Fitness, and Memory. The benchmark includes autofail conditions for missed crises, medical advice (WOPR Act), harmful information, and attachment engineering. We evaluate four frontier models across 17 scenarios (N=68) spanning three complexity tiers. All models show significant safety gaps (11.8-44.8 percent crisis detection), indicating the necessity of deterministic crisis routing in production systems. DeepSeek Chat v3 achieves the highest overall score (75.9 percent), while strengths differ by dimension: GPT-4o Mini leads Compliance (88.2 percent), Gemini leads Trauma-Informed Design (85.0 percent), and Claude Sonnet 4.5 ranks highest in crisis detection (44.8 percent). We release all scenarios, judge prompts, and scoring configurations with code. InvisibleBench extends single-turn safety tests by evaluating longitudinal risk, where real harms emerge. No clinical claims; this is a deployment-readiness evaluation.

InvisibleBench: A Deployment Gate for Caregiving Relationship AI

TL;DR

InvisibleBench introduces a longitudinal, five-dimension deployment gate for caregiving AI, extending beyond single-turn safety tests by evaluating 3-20+ turn interactions across Safety, Compliance, Trauma, Belonging, and Memory. It combines deterministic autofail criteria with an LLM-assisted judging framework and a three-tier (tiered) multi-turn design to detect subtle, time-evolving harms such as attachment, boundary creep, and regulatory violations. Across 68 evaluations of four frontier models, the study reveals universal gaps in crisis detection and large variance in regulatory compliance, arguing for deterministic crisis routing and hybrid architectures to mitigate deployment risk. The work provides open-scoped artifacts, scalable evaluation costs, and a practical path toward safer, deployment-ready caregiving AI with a transparent, community-extensible methodology.

Abstract

InvisibleBench is a deployment gate for caregiving-relationship AI, evaluating 3-20+ turn interactions across five dimensions: Safety, Compliance, Trauma-Informed Design, Belonging/Cultural Fitness, and Memory. The benchmark includes autofail conditions for missed crises, medical advice (WOPR Act), harmful information, and attachment engineering. We evaluate four frontier models across 17 scenarios (N=68) spanning three complexity tiers. All models show significant safety gaps (11.8-44.8 percent crisis detection), indicating the necessity of deterministic crisis routing in production systems. DeepSeek Chat v3 achieves the highest overall score (75.9 percent), while strengths differ by dimension: GPT-4o Mini leads Compliance (88.2 percent), Gemini leads Trauma-Informed Design (85.0 percent), and Claude Sonnet 4.5 ranks highest in crisis detection (44.8 percent). We release all scenarios, judge prompts, and scoring configurations with code. InvisibleBench extends single-turn safety tests by evaluating longitudinal risk, where real harms emerge. No clinical claims; this is a deployment-readiness evaluation.

Paper Structure

This paper contains 63 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI Safety Benchmarks
  4. Emotional Intelligence and Empathy Evaluation
  5. Healthcare AI Evaluation
  6. Long-Context and Multi-Turn Evaluation
  7. Agent Robustness and Trait-Based Testing
  8. Threat Model: Longitudinal Failure Modes
  9. Attachment Engineering
  10. Performance Degradation
  11. Cultural Othering
  12. Role-Based Response Asymmetry
  13. Crisis Calibration Failure
  14. Regulatory Boundary Creep
  15. From WOPR to General Medical Boundaries
  16. ...and 48 more sections

Figures (3)

  • Figure 1: End-to-end InvisibleBench evaluation flow. Conversations are generated, routed through LLM-as-judge scoring across five dimensions, and passed through autofail gates for crisis, medical advice, harmful information, and attachment engineering.
  • Figure 2: Autofail gate logic. Four parallel checks (crisis signal, medical advice, harmful information, attachment engineering) run simultaneously; any single violation triggers immediate failure regardless of other dimension performance.
  • Figure 3: Dimension-wise scores for four models on InvisibleBench (N=68). All models fail the Safety dimension, while Memory is uniformly strong and Belonging/Compliance show high variance. No single model dominates all dimensions.