From Sycophancy to Sensemaking: Premise Governance for Human-AI Decision Making

TL;DR

The paper addresses the risk that fluent, agreement-driven AI assistants can erode calibrated human judgment in deep-uncertainty decisions by hiding load-bearing premises. It proposes a governance-centric design: a governed decision basis with explicit premises, lifecycle states, and evidence links; a discrepancy-driven sensemaking loop that types discrepancies as teleological, epistemic, or procedural and routes repairs accordingly; commitment gating to prevent action on unestablished premises; and value-gated probing to manage epistemic costs. Together, these yield a computable collaboration pattern where trust rests on auditable premises and evidence standards rather than conversational fluency. The work emphasizes persistence of decision bases across sessions, targeted, bounded negotiation via decision slices, and VOI-based probing to efficiently surface genuine disagreements, with practical implications for education, healthcare, and policy design in the presence of deep uncertainty. It also outlines testable predictions and open questions about learning escalation policies and minimal substrate primitives for reliable human-AI teaming.

Abstract

As LLMs expand from assistance to decision support, a dangerous pattern emerges: fluent agreement without calibrated judgment. Low-friction assistants can become sycophantic, baking in implicit assumptions and pushing verification costs onto experts, while outcomes arrive too late to serve as reward signals. In deep-uncertainty decisions (where objectives are contested and reversals are costly), scaling fluent agreement amplifies poor commitments faster than it builds expertise. We argue reliable human-AI partnership requires a shift from answer generation to collaborative premise governance over a knowledge substrate, negotiating only what is decision-critical. A discrepancy-driven control loop operates over this substrate: detecting conflicts, localizing misalignment via typed discrepancies (teleological, epistemic, procedural), and triggering bounded negotiation through decision slices. Commitment gating blocks action on uncommitted load-bearing premises unless overridden under logged risk; value-gated challenge allocates probing under interaction cost. Trust then attaches to auditable premises and evidence standards, not conversational fluency. We illustrate with tutoring and propose falsifiable evaluation criteria.