OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding

TL;DR

OctoBench addresses the need to evaluate scaffold-aware instruction following in repository-grounded coding by introducing a long-horizon benchmark with 34 environments, 217 tasks, and 7,098 binary checklist items. It pairs executable task environments with multi-source instruction sources, an observation harness, and an automated LLM-based judge to disentangle task solving from rule adherence. The study reveals a substantial gap between per-check compliance and end-to-end success, substantial variation across instruction categories, and limited cross-scaffold robustness, while showing that external supervisory signals can drive improvements. The work provides a reproducible benchmarking pipeline and actionable insights to guide the development of more robust scaffold-aware coding agents.

Abstract

Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are heterogeneous and persist across interactions. To fill this gap, we introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding. OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items. To disentangle solving the task from following the rules, we provide an automated observation-and-scoring toolkit that captures full trajectories and performs fine-grained checks. Experiments on eight representative models reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation that explicitly targets heterogeneous instruction following. We release the benchmark to support reproducible benchmarking and to accelerate the development of more scaffold-aware coding agents.

Figures (4)

• Figure 1: Overview of OctoBench. OctoBench evaluates instruction following in realistic agentic coding by combining heterogeneous, persistent instruction sources with a scaffold that interacts with an executable environment, while an observation harness records trajectories. These trajectories are then mapped to an instance-specific binary checklist that operationalizes verifiable constraints across all evidenced sources, and are scored via an LLM-as-a-judge to produce fine-grained metrics, disentangling solving the task from following the rules.
• Figure 2: OctoBench dataset construction pipeline. Starting from raw instruction-carrying materials, human annotators curate executable task and expend the curated queries(\ref{['sec:datasets_task_construction']}). For each task, we execute a reference agent in the packaged environment to collect trajectories, and use LLM-assisted checklist generation followed by joint human–LLM review (\ref{['sec:datasets_checklist_construction']}). Each released instance bundles the task and checklist.
• Figure 3: Analysis of ISR trends across varying interaction turns.
• Figure 4: Rank stability analysis across three distinct judge models. The x-axis represents the judge models, and the y-axis represents the ranking of the evaluated models, where rankings are computed by ISR score.