Assessing the Feasibility of Selective Instrumentation for Runtime Code Coverage in Large C++ Game Engines

TL;DR

AAA game studios struggle to collect code coverage without incurring prohibitive overhead. The authors propose a commit-level selective instrumentation framework built on LLVM Clang that instruments only functions touched by a commit, preserving actionable coverage while keeping build and runtime costs low. They implement an end-to-end pipeline including profile list extraction from diffs, mangled-name resolution via a Clang AST, integration with a game wrapper and coverage reporting, and an instrumentation API. Through experiments in compilation time, runtime performance, and test stability, they show that commit-based selective instrumentation can maintain frame rates and avoid automated test failures while enabling coverage of large C++ game engines. The work demonstrates practical feasibility of larger-scale coverage in production workflows and offers a path to broader adoption in AAA game development.

Abstract

Code coverage is a valuable guide for testing, but in AAA games the overhead of instrumentation conflicts with strict performance requirements and can destabilize automated tests. We propose and assess a selective instrumentation approach tailored to large game engines written in \texttt{C++}, which reduces the scope of instrumentation while preserving relevant coverage data to developer commits. Our framework integrates into an industrial game testing pipeline, enabling developers to receive immediate coverage feedback on tests run against their changes. The compilation overhead of our approach is minimal, allowing instrumentation of over 2,000 commits before doubling build time. In performance evaluations, even the worst-case scenario maintains frame rates above 50\% of the non-instrumented baseline. Across two production test suites maintained by our industry partner, our framework caused no automated test failures, avoiding the instability observed under full instrumentation. Our work shows that commit-level or build-level coverage of large \texttt{C++} game engines can be achieved with minimal overhead and without compromising test stability.

Figures (6)

• Figure 1: Different types of instrumentation are added at different points in the compilation process, either before or during optimization.
• Figure 2: Framework overview
• Figure 3: Extracting a profile list from a commit
• Figure 4: Overhead introduced by profile list extraction
• Figure 5: Relationship between $\mathrm{IFR}$ and $t_{\mathrm{CPU}}$ at different scales.