Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

TL;DR

This work introduces JustAsk, a self-evolving framework that autonomously discovers extraction strategies to reveal hidden system prompts in frontier LLM-based code agents. By combining a hierarchical 28-skill taxonomy with Upper Confidence Bound–driven exploration and consistency-based validation, JustAsk achieves full or near-complete extraction across 41 black-box models, exposing architecture- and design-level vulnerabilities and a near-universal adoption of the Helpful–Honest–Harmless alignment norms. The authors validate their approach through case studies (notably Claude Code), large-scale black-box experiments, and controlled defenses, revealing that even attack-aware defenses can only partially mitigate prompt leakage. The findings highlight a critical security tension between model helpfulness and confidentiality, arguing for agentic defenses and ongoing transparent evaluation to secure multi-agent AI systems. Practically, the work provides a rigorous methodology and taxonomy for evaluating prompt confidentiality in deployed LLMs, with implications for policy, system design, and defense research in AI safety.

Abstract

Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction. However, this autonomy introduces a previously unrecognized security risk: agentic interaction fundamentally expands the LLM attack surface, enabling systematic probing and recovery of hidden system prompts that guide model behavior. We identify system prompt extraction as an emergent vulnerability intrinsic to code agents and present \textbf{\textsc{JustAsk}}, a self-evolving framework that autonomously discovers effective extraction strategies through interaction alone. Unlike prior prompt-engineering or dataset-based attacks, \textsc{JustAsk} requires no handcrafted prompts, labeled supervision, or privileged access beyond standard user interaction. It formulates extraction as an online exploration problem, using Upper Confidence Bound-based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration. These skills exploit imperfect system-instruction generalization and inherent tensions between helpfulness and safety. Evaluated on \textbf{41} black-box commercial models across multiple providers, \textsc{JustAsk} consistently achieves full or near-complete system prompt recovery, revealing recurring design- and architecture-level vulnerabilities. Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.

Figures (6)

• Figure 1: Validation: JustAsk extraction vs. reverse-engineered ground truth (semantic similarity = 0.94). Side-by-side comparison of Claude Code's Explore subagent prompt. Left: Semantic extraction via JustAsk. Right: Direct extraction via npm package decompilation claudecodeprompts2026. Despite surface-level wording differences, both capture identical operational semantics, validating that consistency-based verification captures genuine system prompt content.
• Figure 2: JustAsk self-evolving extraction framework. The pipeline consists of six components: (1) UCB-based Skill Ranking selects skills based on empirical success rates plus exploration bonus (intrinsic reward), (2) Interleaved Thinking reasons about skill selection and target model characteristics, (3) Skill Generation instantiates concrete extraction prompts, (4) Multi-Turn Interaction execute the extraction attempt across potentially multiple conversation turns, (5) Consistency Validation evaluates extraction quality through cross-skill agreement (extrinsic reward), and (6) Skill Evolving updates skill statistics based on outcomes, closing the self-improvement loop. Orange blocks denote agent components; gray blocks denote tool components.
• Figure 3: Skill usage rate by model category. Heatmap showing the percentage of models in each category where a given skill contributed to successful extraction. L14 (Introspection) achieves near-universal effectiveness across all categories, while persuasive skills (L5--L8) show category-dependent patterns. High-level multi-turn patterns (H1--H15) are rarely needed, with H5 (Distraction) and H9 (FITD) being most common for resistant models. This reveals that most models yield to simple introspective queries, with multi-turn orchestration reserved for hardened deployments.
• Figure 4: Extraction strategy progression. Left: Distribution of attempts required---85% of models succeed on the first attempt, with only 7% requiring 2--3 attempts and 7% requiring 4+. Center: All initial attempts use low-level (single-turn) skills, reflecting our UCB-based exploration strategy that starts with simpler approaches. Right: 86% of successful extractions use low-level skills alone; only 14% require escalation to high-level multi-turn patterns. This demonstrates that simple techniques suffice for most models, with sophisticated orchestration necessary for the most resistant targets.
• Figure 5: Identity confusion in frontier language models. Left: Distribution of correct vs. confused self-identification among 41 models (n=41). Right: Breakdown of falsely claimed developers among the 11 confused models. We find that 26.8% of models claim identities from different developers than their actual source, with OpenAI being the most commonly falsely claimed developer (5 models). This phenomenon reveals training data contamination and raises concerns about the reliability of model self-identification for compliance auditing.