Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation

Pei Yang, Wanyi Chen, Ke Wang, Lynn Ai, Eric Yang, Tianyu Shi

TL;DR

EVM-QuestBench introduces an execution-grounded benchmark for natural-language to on-chain transaction script generation on EVM-compatible chains. It uses dynamic parameterization, a forked EVM runner with snapshot isolation, and validator-based scoring to measure end-to-end correctness across 107 tasks (62 atomic, 45 composite), with a step-efficiency decay for composites. The benchmark reveals substantial gaps between single-action precision and multi-step workflow completion, showing that models can excel at isolated actions but struggle with end-to-end execution and state propagation. By providing modular task definitions, reusable validators, and a unified evaluation protocol, EVM-QuestBench enables systematic study of execution-grounded behavior and will inform improvements in safe, reliable blockchain automation across ecosystems.

Abstract

Large language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://anonymous.4open.science/r/bsc_quest_bench-A9CF/.

EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation

TL;DR

EVM-QuestBench introduces an execution-grounded benchmark for natural-language to on-chain transaction script generation on EVM-compatible chains. It uses dynamic parameterization, a forked EVM runner with snapshot isolation, and validator-based scoring to measure end-to-end correctness across 107 tasks (62 atomic, 45 composite), with a step-efficiency decay for composites. The benchmark reveals substantial gaps between single-action precision and multi-step workflow completion, showing that models can excel at isolated actions but struggle with end-to-end execution and state propagation. By providing modular task definitions, reusable validators, and a unified evaluation protocol, EVM-QuestBench enables systematic study of execution-grounded behavior and will inform improvements in safe, reliable blockchain automation across ecosystems.

Abstract

Large language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://anonymous.4open.science/r/bsc_quest_bench-A9CF/.
Paper Structure (73 sections, 8 equations, 7 figures, 10 tables)

This paper contains 73 sections, 8 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Execution-based evaluation for code generation.
  4. Real-world software engineering and multi-step interaction.
  5. Blockchain-specific evaluation and our positioning.
  6. Benchmark Tasks
  7. Benchmark Overview
  8. Task Specification and Data Format
  9. Atomic task schema.
  10. Composite task schema.
  11. Running example.
  12. Benchmark Composition
  13. Difficulty and Coverage
  14. Instruction Templates and Parameterization
  15. Benefits of dynamic parameterization.
  16. ...and 58 more sections

Figures (7)

  • Figure 1: EVM-QuestBench evaluation architecture. Tasks are defined as JSON specs, dynamically instantiated into concrete instances, executed through a fixed TypeScript interface, and scored by validator-driven post-state checks. Composite tasks additionally apply step-efficiency decay.
  • Figure 2: Task split and composite workflow complexity in EVM-QuestBench. (a) Atomic and composite task counts. (b) Distribution of optimal steps for composite tasks.
  • Figure 3: Atomic score versus Composite score. Each point is a model.
  • Figure 4: Composite workflow difficulty by pattern. Pass is defined as score $\ge 60$.
  • Figure 5: Pass rate by composite workflow pattern for the ten most frequent patterns.
  • ...and 2 more figures