CounterPoint: Using Hardware Event Counters to Refute and Refine Microarchitectural Assumptions (Extended Version)

TL;DR

CounterPoint automates the evaluation of microarchitectural hypotheses against large, noisy hardware event counter (HEC) data by representing expert ideas as μpath Decision Diagrams (μDDs) and deriving a model cone that captures all feasible HEC combinations. It then constructs tight counter confidence regions to mitigate multiplexing noise and uses linear programming feasibility testing to confirm or refute μDD-derived constraints, enabling iterative model refinement. The Haswell MMU case study demonstrates how CounterPoint uncovers undocumented features such as a TLB prefetcher, page-walk merging, and aborted walks, illustrating the framework's ability to reveal subtle hardware behaviors beyond traditional analysis. The approach scales to many counters, reduces reliance on manual constraint derivation, and provides a practical path toward forging trustworthy, data-driven microarchitectural models for complex architectures.

Abstract

Hardware event counters offer the potential to reveal not only performance bottlenecks but also detailed microarchitectural behavior. In practice, this promise is undermined by their vague specifications, opaque designs, and multiplexing noise, making event counter data hard to interpret. We introduce CounterPoint, a framework that tests user-specified microarchitectural models - expressed as $μ$path Decision Diagrams - for consistency with performance counter data. When mismatches occur, CounterPoint pinpoints plausible microarchitectural features that could explain them, using multi-dimensional counter confidence regions to mitigate multiplexing noise. We apply CounterPoint to the Haswell Memory Management Unit as a case study, shedding light on multiple undocumented and underdocumented microarchitectural behaviors. These include a load-store queue-side TLB prefetcher, merging page table walkers, abortable page table walks, and more. Overall, CounterPoint helps experts reconcile noisy hardware performance counter measurements with their mental model of the microarchitecture - uncovering subtle, previously hidden hardware features along the way.

Figures (10)

• Figure 1: The rapid growth of HECs has increased manual effort to construct and compose model constraints, and amplified multiplexing noise that obscures constraint violations. (a) The blue line shows the number of documented HEC 'names', assuming a single core. The red line shows the number of 'addressable' events after accounting for per-core replication and the conservative removal of events that, while still documented (and potentially informative), have been deprecated by the vendor. Each red data point represents a microarchitecture paired with its typical core count in a server system. This graph shows only documented events and does not include the thousands of additional undocumented HECs identified in recent work ExplorationOfHiddenPmuEvents. (b) The number of model constraints grows superlinearly with the number of HECs (our x-axis shows increasing HEC count for an Intel Haswell MMU, in steps associated with all the HECs in a logical group; e.g., 10 HECs for L2 TLB events) and worsens significantly when including hypothetical HECs across all MMU caches (shown in green). (c) For a representative model constraint on the Intel Haswell MMU ((1) in Table \ref{['tab:feasibility-constraints']}), we show that as measurement noise increases—both overall and for individual HECs—it becomes impossible to determine whether the model constraint is violated with 99% confidence once 19 HECs are active. Here, noise is defined as the standard deviation in the observed HEC values.
• Figure 2: CounterPoint automatically determines the feasibility of a microarchitectural model against HEC data. Models are described using a DSL and transformed into a $\mu$path Decision Diagram ($\mu$DD), which is analyzed to determine the model cone (the set of model constraints). Counter confidence regions are constructed for each observation to handle multiplexing noise. Observations are tested against all model constraints simultaneously. CounterPoint's counter confidence region bounds are sharper than other approaches, enabling more violations to be identified, and thereby enabling more opportunities to refine the expert's microarchitectural assumptions. CounterPoint effortlessly supports dozens of HECs and constraints.
• Figure 3: The ability of HECs to test assumptions depends on their number and semantics, shown here pictorially. The orange regions represent points which satisfy all model constraints; the blue dot represents an observation; the red and green boxes represent two alternative constructions of counter confidence regions. Model constraints correspond to edges in 2D or faces in 3D. (a) Consider a model cone constructed from the three HECs shown. These counters imply three constraints: $\texttt{load.ret\_stlb\_miss}\xspace \leq \texttt{load.walk\_done}\xspace$ because each retired STLB miss must correspond to a unique, successfully completed page table walk; $\texttt{load.ret\_stlb\_miss}\xspace \leq \texttt{load.causes\_walk}\xspace$ because each retired STLB miss must trigger exactly one page table walk; and $\texttt{load.walk\_done}\xspace \leq \texttt{load.causes\_walk}\xspace$ because only a subset of initiated page table walks ultimately complete. The first two inequalities rely on the assumption that STLB misses are never merged. Using all three HECs clearly exposes a violation of these constraints, indicating a flaw in the expert’s mental model. (b) All three HECs were required to detect this flaw; removing $\texttt{load.walk\_done}\xspace$ eliminates the second and third constraints, making the model violation undetectable. (c) Simply substituting $\texttt{load.walk\_done}\xspace$ with $\texttt{load.pde\$\_miss}\xspace$ (or any other counter) is insufficient, because the semantics of each counter matter. Using this alternative counter adds the constraint $\texttt{load.pde\$\_miss}\xspace \leq \texttt{load.causes\_walk}\xspace$, but this constraint still fails to reveal the model violation. (d) Counter confidence regions replace point observations with value ranges; exploiting correlations yields tighter bounds than assuming independence.
• Figure 4: A $\mu$DD encodes a set of microarchitectural execution paths ($\mu$paths). Each $\mu$path describes a set of events per $\mu$op.
• Figure 5: The model cone is determined purely by the $\mu$path counter signatures (Figure a). Testing observations for inclusion in the model cone is complicated by noise, which can cause observations to spuriously appear infeasible (Figure b). CounterPoint handles noise by constructing confidence regions at the 99% confidence level (Figure c). The counter confidence region is an ellipsoid which CounterPoint approximates by its bounding box, enabling a linear programming formulation. The scale and orientation of the confidence region is determined by (i) the confidence level and (ii) correlations in the observed data. $\lambda_k$ and $\vec{e_k}$ denote the $k$th eigenvalue/eigenvector of the estimated covariance matrix.