Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles

Srinath Srinivasan, Tim Menzies, Marcelo D'Amorim

TL;DR

SmartOracle presents an agentic framework that decomposes the differential fuzzing triage task into specialized LLM sub-agents to filter noise and identify true JavaScript specification violations. By combining discrepancy analysis, specification grounding, and terminal verification with a semi-supervised ground-truth propagation method, it achieves $0.84$ recall with $18\%$ false positives, while delivering $4\times$ faster analysis and $10\times$ lower API costs than a sequential LRM baseline. In active fuzzing campaigns, it uncovered previously unknown, specification-level bugs across V8, JavaScriptCore, GraalJS, and related engines, with several confirmed or fixed upstream. The approach demonstrates that agented architectures, when paired with semi-supervised evaluation, can meaningfully scale and improve the reliability of differential oracles in complex software systems, with potential applicability to other domains beyond JavaScript engines.

Abstract

Differential fuzzers detect bugs by executing identical inputs across distinct implementations of the same specification, such as JavaScript interpreters. Validating the outputs requires an oracle and for differential testing of JavaScript, these are constructed manually, making them expensive, time-consuming, and prone to false positives. Worse, when the specification evolves, this manual effort must be repeated. Inspired by the success of agentic systems in other SE domains, this paper introduces SmartOracle. SmartOracle decomposes the manual triage workflow into specialized Large Language Model (LLM) sub-agents. These agents synthesize independently gathered evidence from terminal runs and targeted specification queries to reach a final verdict. For historical benchmarks, SmartOracle achieves 0.84 recall with an 18% false positive rate. Compared to a sequential Gemini 2.5 Pro baseline, it improves triage accuracy while reducing analysis time by 4$\times$ and API costs by 10$\times$. In active fuzzing campaigns, SmartOracle successfully identified and reported previously unknown specification-level issues across major engines, including bugs in V8, JavaScriptCore, and GraalJS. The success of SmartOracle's agentic architecture on Javascript suggests it might be useful other software systems- a research direction we will explore in future work.

SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles

TL;DR

SmartOracle presents an agentic framework that decomposes the differential fuzzing triage task into specialized LLM sub-agents to filter noise and identify true JavaScript specification violations. By combining discrepancy analysis, specification grounding, and terminal verification with a semi-supervised ground-truth propagation method, it achieves recall with false positives, while delivering faster analysis and lower API costs than a sequential LRM baseline. In active fuzzing campaigns, it uncovered previously unknown, specification-level bugs across V8, JavaScriptCore, GraalJS, and related engines, with several confirmed or fixed upstream. The approach demonstrates that agented architectures, when paired with semi-supervised evaluation, can meaningfully scale and improve the reliability of differential oracles in complex software systems, with potential applicability to other domains beyond JavaScript engines.

Abstract

Differential fuzzers detect bugs by executing identical inputs across distinct implementations of the same specification, such as JavaScript interpreters. Validating the outputs requires an oracle and for differential testing of JavaScript, these are constructed manually, making them expensive, time-consuming, and prone to false positives. Worse, when the specification evolves, this manual effort must be repeated. Inspired by the success of agentic systems in other SE domains, this paper introduces SmartOracle. SmartOracle decomposes the manual triage workflow into specialized Large Language Model (LLM) sub-agents. These agents synthesize independently gathered evidence from terminal runs and targeted specification queries to reach a final verdict. For historical benchmarks, SmartOracle achieves 0.84 recall with an 18% false positive rate. Compared to a sequential Gemini 2.5 Pro baseline, it improves triage accuracy while reducing analysis time by 4 and API costs by 10. In active fuzzing campaigns, SmartOracle successfully identified and reported previously unknown specification-level issues across major engines, including bugs in V8, JavaScriptCore, and GraalJS. The success of SmartOracle's agentic architecture on Javascript suggests it might be useful other software systems- a research direction we will explore in future work.
Paper Structure (44 sections, 5 equations, 10 figures, 6 tables)

This paper contains 44 sections, 5 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Motivation
  4. Differential Fuzzing
  5. LLM Agents
  6. Semi-supervised label propagation
  7. Differences from previous work
  8. Experimental Methodology
  9. Datasets
  10. Historical Differential Fuzzing Data
  11. Manually Labeled Fuzzing Data
  12. Sequential LLM Prompting
  13. Rationale for model architecture selection
  14. Agent Design
  15. Orchestrator
  16. ...and 29 more sections

Figures (10)

  • Figure 1: Example of benign output differences across three JavaScript engines. While the outputs differ textually, all are compliant with the ECMA-262 specification.
  • Figure 2: Intersecting related works. SmartOracle is the first to discuss all the related concepts
  • Figure 3: Overview of how a single test file is processed and the differential findings are analyzed by SmartOracle
  • Figure 4: Sequentially prompted LRM setup
  • Figure 5: Overview of SmartOracle, its sub-agents, and tools.
  • ...and 5 more figures