Compass vs Railway Tracks: Unpacking User Mental Models for Communicating Long-Horizon Work to Humans vs. AI
Savvas Petridis, Michael Xieyang Liu, Alexander J. Fiannaca, Carrie J. Cai, Michael Terry
TL;DR
The paper investigates how users communicate long-horizon tasks to AI versus human colleagues, introducing the Compass vs Railway Tracks framework to capture a fundamental dichotomy: humans receive high-level compass guidance that enables flexible exploration, while AI receives rigid, exhaustive railway-like instructions to constrain execution. Through a qualitative study with 16 professionals writing paired specifications, the authors show that AI-facing specs tend to be longer and more prescriptive, reflecting beliefs that current models struggle to infer intent, prioritize tasks, or self-monitor. They also reveal a desire for a hybrid AI collaborator that blends human-typical critical thinking with AI-scale efficiency, governed by proactive dialogue and minimal social overhead. The authors propose three design implications—AI-generated rough drafts, end-to-end test runs, and intelligent check-ins—to realign AI with human goal setting and verification, advancing AI from a passive instruction follower to a reliable long-horizon partner for ambiguous problems.
Abstract
As AI systems (foundation models, agentic systems) grow increasingly capable of operating for minutes or hours at a time, users' prompts are transforming into highly detailed, elaborate specifications for the AI to autonomously work on. While interactive prompting has been extensively studied, comparatively less is known about how people communicate specifications for these types of long-horizon tasks. In a qualitative study in which 16 professionals drafted specifications for both a human colleague and an AI, we found a core divergence in how people specified problems to people versus AI: people approached communication with humans as providing a "compass", offering high-level intent to encourage flexible exploration. In contrast, communication with AI resembled painstakingly laying down "railway tracks": rigid, exhaustive instructions to minimize ambiguity and deviation. This strategy was driven by a perception that current AI has limited ability to infer intent, prioritize, and make judgments on its own. When envisioning an idealAI collaborator, users expressed a desire for a hybrid between current AI and human colleagues: a collaborator that blends AI's efficiency and large context window with the critical thinking and agency of a human colleague. We discuss design implications for future AI systems, proposing that they align on outcomes through generated rough drafts, verify feasibility via end-to-end "test runs," and monitor execution through intelligent check-ins, ultimately transforming AI from a passive instruction-follower into a reliable collaborator for ambiguous, long-horizon problems.