Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COMET: Neural Cost Model Explanation Framework

Isha Chaudhary, Alex Renda, Charith Mendis, Gagandeep Singh

TL;DR

The empirical findings show an inverse correlation between the prediction errors of Ithemal and uiCA and the granularity of basic block features in COMET's explanations for them, thus indicating potential reasons for the higher error of Ithemal with respect to uiCA.

Abstract

Cost models predict the cost of executing given assembly code basic blocks on a specific microarchitecture. Recently, neural cost models have been shown to be fairly accurate and easy to construct. They can replace heavily engineered analytical cost models used in mainstream compiler workflows. However, their black-box nature discourages their adoption. In this work, we develop the first framework, COMET, for generating faithful, generalizable, and intuitive explanations for neural cost models. We generate and compare COMET's explanations for the popular neural cost model, Ithemal against those for an accurate CPU simulation-based cost model, uiCA. Our empirical findings show an inverse correlation between the prediction errors of Ithemal and uiCA and the granularity of basic block features in COMET's explanations for them, thus indicating potential reasons for the higher error of Ithemal with respect to uiCA.

COMET: Neural Cost Model Explanation Framework

TL;DR

The empirical findings show an inverse correlation between the prediction errors of Ithemal and uiCA and the granularity of basic block features in COMET's explanations for them, thus indicating potential reasons for the higher error of Ithemal with respect to uiCA.

Abstract

Cost models predict the cost of executing given assembly code basic blocks on a specific microarchitecture. Recently, neural cost models have been shown to be fairly accurate and easy to construct. They can replace heavily engineered analytical cost models used in mainstream compiler workflows. However, their black-box nature discourages their adoption. In this work, we develop the first framework, COMET, for generating faithful, generalizable, and intuitive explanations for neural cost models. We generate and compare COMET's explanations for the popular neural cost model, Ithemal against those for an accurate CPU simulation-based cost model, uiCA. Our empirical findings show an inverse correlation between the prediction errors of Ithemal and uiCA and the granularity of basic block features in COMET's explanations for them, thus indicating potential reasons for the higher error of Ithemal with respect to uiCA.
Paper Structure (32 sections, 1 theorem, 11 equations, 10 figures, 3 tables, 1 algorithm)

This paper contains 32 sections, 1 theorem, 11 equations, 10 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Overview and motivating example
  4. Formalizing cost model explanations
  5. Ideal query-only explanation
  6. Practical query-only explanations
  7. COMET: Neural Cost Model Explanation Framework
  8. Extracting block features
  9. Efficiently computing explanations
  10. Evaluation
  11. Accuracy-based evaluation of COMET
  12. Precision and coverage evaluation
  13. Evaluating utility of COMET
  14. Case studies
  15. Discussion and future work
  16. ...and 17 more sections

Key Result

Theorem 1

$\Pi$ is a monotonically decreasing function.

Figures (10)

  • Figure 1: COMET is given a cost model $\mathcal{M}$ and a basic block $\beta$ as input. COMET identifies the features of $\beta$ that explain the prediction $\mathcal{M}(\beta)$. COMET first converts $\beta$ to a multigraph $\mathcal{G}$ in (a). $\mathcal{G}$ has the instructions and data dependencies of $\beta$ as its vertices and edges respectively. The features $\Hat{\mathcal{P}}$ of $\beta$, are then extracted from $\mathcal{G}$ in (b). Sets of these features are fed into COMET's perturbation algorithm $\Gamma$ (c) that generates several perturbations that preserve the features in the corresponding retained input feature set $\mathcal{F}$. COMET obtains the predictions of cost model $\mathcal{M}$ for each perturbed basic block, which are then used for the estimation of the precision and coverage of a feature set $\mathcal{F}$. $\mathcal{F}$ having precision higher than $(1-\delta)$ with maximum coverage is identified by the precision and coverage optimizer in (d) and is output as COMET's explanation for $\mathcal{M}(\beta)$.
  • Figure 2: Input basic block
  • Figure 3: Perturbed basic block
  • Figure 4: Variation of Mean Absolute Percentage Error (MAPE) in Ithemal and uiCA alongside variation in the % of explanations consisting: number of instructions $\eta$, specific instructions $inst$ and data dependencies $\delta$. Figure (a): Haswell, (b): Skylake
  • Figure 5: Variation of Mean Absolute Percentage Error (MAPE) in Ithemal and uiCA with the % of explanations consisting: number of instructions $\eta$, instructions $inst$ and data dependencies $\delta$. BHive sources: (a) Clang, (b) OpenBLAS
  • ...and 5 more figures

Theorems & Definitions (2)

  • Theorem 1
  • proof