Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpDiffer: LLM-Assisted Opcode-Level Differential Testing of Ethereum Virtual Machine

Jie Ma, Ningyu He, Jinwen Xi, Mingzhe Xing, Haoyu Wang, Ying Gao, Yinliang Yue

TL;DR

OpDiffer tackles the underexplored security of the Ethereum Virtual Machine by introducing opcode-level differential testing guided by LLMs and static analysis. It generates semantically valid, diverse test inputs via an ICFG-informed seed-and-mutate pipeline and couples this with automated bug identification and root-cause localization across multiple EVM implementations. In nine EVMs, OpDiffer identifies 26 bugs (22 confirmed; 3 CNVD IDs) and achieves significant code-coverage improvements relative to baselines, while real-world contract analysis suggests a non-negligible exposure of deployed contracts to these bugs (up to 7.21% in the Etherscan dataset). The work demonstrates the practical impact of opcode-level testing for improving EVM reliability and guiding targeted fixes in the Ethereum ecosystem.

Abstract

As Ethereum continues to thrive, the Ethereum Virtual Machine (EVM) has become the cornerstone powering tens of millions of active smart contracts. Intuitively, security issues in EVMs could lead to inconsistent behaviors among smart contracts or even denial-of-service of the entire blockchain network. However, to the best of our knowledge, only a limited number of studies focus on the security of EVMs. Moreover, they suffer from 1) insufficient test input diversity and invalid semantics; and 2) the inability to automatically identify bugs and locate root causes. To bridge this gap, we propose OpDiffer, a differential testing framework for EVM, which takes advantage of LLMs and static analysis methods to address the above two limitations. We conducted the largest-scale evaluation, covering nine EVMs and uncovering 26 previously unknown bugs, 22 of which have been confirmed by developers and three have been assigned CNVD IDs. Compared to state-of-the-art baselines, OpDiffer can improve code coverage by at most 71.06%, 148.40% and 655.56%, respectively. Through an analysis of real-world deployed Ethereum contracts, we estimate that 7.21% of the contracts could trigger our identified EVM bugs under certain environmental settings, potentially resulting in severe negative impact on the Ethereum ecosystem.

OpDiffer: LLM-Assisted Opcode-Level Differential Testing of Ethereum Virtual Machine

TL;DR

OpDiffer tackles the underexplored security of the Ethereum Virtual Machine by introducing opcode-level differential testing guided by LLMs and static analysis. It generates semantically valid, diverse test inputs via an ICFG-informed seed-and-mutate pipeline and couples this with automated bug identification and root-cause localization across multiple EVM implementations. In nine EVMs, OpDiffer identifies 26 bugs (22 confirmed; 3 CNVD IDs) and achieves significant code-coverage improvements relative to baselines, while real-world contract analysis suggests a non-negligible exposure of deployed contracts to these bugs (up to 7.21% in the Etherscan dataset). The work demonstrates the practical impact of opcode-level testing for improving EVM reliability and guiding targeted fixes in the Ethereum ecosystem.

Abstract

As Ethereum continues to thrive, the Ethereum Virtual Machine (EVM) has become the cornerstone powering tens of millions of active smart contracts. Intuitively, security issues in EVMs could lead to inconsistent behaviors among smart contracts or even denial-of-service of the entire blockchain network. However, to the best of our knowledge, only a limited number of studies focus on the security of EVMs. Moreover, they suffer from 1) insufficient test input diversity and invalid semantics; and 2) the inability to automatically identify bugs and locate root causes. To bridge this gap, we propose OpDiffer, a differential testing framework for EVM, which takes advantage of LLMs and static analysis methods to address the above two limitations. We conducted the largest-scale evaluation, covering nine EVMs and uncovering 26 previously unknown bugs, 22 of which have been confirmed by developers and three have been assigned CNVD IDs. Compared to state-of-the-art baselines, OpDiffer can improve code coverage by at most 71.06%, 148.40% and 655.56%, respectively. Through an analysis of real-world deployed Ethereum contracts, we estimate that 7.21% of the contracts could trigger our identified EVM bugs under certain environmental settings, potentially resulting in severe negative impact on the Ethereum ecosystem.

Paper Structure

This paper contains 28 sections, 10 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Ethereum Virtual Machine
  4. Differential Testing
  5. Challenges & Solution
  6. Motivating Example & Challenges
  7. Solution
  8. Approach
  9. Overview
  10. LLM-Assisted Test Input Generation
  11. Step I: Opcode Seeds Construction.
  12. Step II: Specification-Based ICFG Extraction.
  13. Step III: Control-Flow-Oriented Mutation.
  14. Step IV: Argument-Oriented Mutation.
  15. Differential Testing
  16. ...and 13 more sections

Figures (10)

  • Figure 1: A motivating example about the bug of the EXP opcode in SealEVM.
  • Figure 2: An example for illustrating a semantically-invalid case.
  • Figure 3: The workflow overview of OpDiffer.
  • Figure 4: Prompt for generating seed generator of the BYTE opcode.
  • Figure 5: A generated seed generator for the BYTE opcode.
  • ...and 5 more figures