Taming Scylla: Understanding the multi-headed agentic daemon of the coding seas

TL;DR

Scylla introduces a model-agnostic evaluation framework for agentic coding tools that uses seven tiered ablations (T0–T6) to quantify how prompts, skills, tools, and agent hierarchies impact performance and cost, with Cost-of-Pass ($CoP$) as the central metric. In a dryrun using Claude Code and three judges, the study shows quality saturates across tiers on a simple Hello World task, while CoP varies by up to 3.8x, highlighting economies of selective feature loading (e.g., T5 hybrid) over maximal configurations. The framework emphasizes reproducibility through git worktrees and checkpoints, and demonstrates how tiered architecture and multi-judge scoring can dissect cost-accuracy trade-offs in real CLI-tool environments. The findings suggest that architectural complexity does not universally improve outcomes and advocate scaling Scylla to broader tasks and cross-vendor comparisons to establish practical deployment guidance for automated coding tools.

Abstract

LLM-based tools are automating more software development tasks at a rapid pace, but there is no rigorous way to evaluate how different architectural choices -- prompts, skills, tools, multi-agent setups -- materially affect both capability and cost. This paper introduces Scylla, an evaluation framework for benchmarking agentic coding tools through structured ablation studies that uses seven testing tiers (T0-T6) progressively adding complexity to isolate what directly influences results and how. The key metric is Cost-of-Pass (CoP): the expected dollar cost to get one correct solution, which directly quantifies the trade-off between complexity and efficiency. The framework is model-agnostic, designed to work with any CLI tool; this paper demonstrates it with Claude Sonnet 4.5, using multiple LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) from the same vendor for evaluation consensus, where judges score results using direct tests, human-designed LLM-evaluated rubrics, and qualitative assessment. The result is a reproducible framework that quantifies trade-offs between agent complexity and actual outcomes, suggesting that architectural complexity does not always improve quality.

Figures (7)

• Figure 1: System architecture showing the seven-layer pipeline. Data flows top-to-bottom through: orchestration, workspace isolation, configuration injection, checkpoint management, CLI execution, LLM evaluation, and metrics analysis.
• Figure 2: Execution pipeline for a single evaluation run. Each step produces inputs for the next, enabling reproducibility through saved artifacts at every boundary.
• Figure 3: Tier dependency graph and parallel execution model. T0-T4 execute in parallel (Phase 1). T5 waits for T0-T4 completion and merges their best configurations (Phase 2). T6 waits for T5 and enables all features (Phase 3).
• Figure 4: Token distribution by tier and type. Stacked bar chart showing the breakdown of input, output, cache create, and cache read tokens across T0--T6. Cache read tokens dominate (79--95%), consistent with prompt caching efficacy. However, T6's 218K cache reads versus T0's 113K illustrate the Token Efficiency Chasm, where architectural enhancements double token consumption without quality gains.
• Figure 5: Per-judge scoring variance across tiers. Box plots showing score distributions for each judge model (Opus 4.5, Sonnet 4.5, Haiku 4.5) faceted by tier. Opus exhibits the tightest distribution (most conservative), Haiku the widest (most generous), revealing systematic inter-judge bias that affects aggregate score reliability.