Interpretable Concept-Based Memory Reasoning

TL;DR

Concept-based Memory Reasoner is introduced, a novel CBM designed to provide a human-understandable and provably-verifiable task prediction process and achieves better accuracy-interpretability trade-offs to state-of-the-art CBMs.

Abstract

The lack of transparency in the decision-making processes of deep learning systems presents a significant challenge in modern artificial intelligence (AI), as it impairs users' ability to rely on and verify these systems. To address this challenge, Concept Bottleneck Models (CBMs) have made significant progress by incorporating human-interpretable concepts into deep learning architectures. This approach allows predictions to be traced back to specific concept patterns that users can understand and potentially intervene on. However, existing CBMs' task predictors are not fully interpretable, preventing a thorough analysis and any form of formal verification of their decision-making process prior to deployment, thereby raising significant reliability concerns. To bridge this gap, we introduce Concept-based Memory Reasoner (CMR), a novel CBM designed to provide a human-understandable and provably-verifiable task prediction process. Our approach is to model each task prediction as a neural selection mechanism over a memory of learnable logic rules, followed by a symbolic evaluation of the selected rule. The presence of an explicit memory and the symbolic evaluation allow domain experts to inspect and formally verify the validity of certain global properties of interest for the task prediction process. Experimental results demonstrate that CMR achieves better accuracy-interpretability trade-offs to state-of-the-art CBMs, discovers logic rules consistent with ground truths, allows for rule interventions, and allows pre-deployment verification.

Figures (8)

• Figure 1: Probabilistic graphical model of CMR
• Figure 2: Example prediction of CMR with a rulebook of two rules and three concepts (i.e. $\mathit{red}\,(R)$, $\mathit{square}\,(S)$, $\mathit{table}\,(T)$). In this figure, we sample ($\sim$) for clarity, but in practice, we compute every probability exactly. Every black box is implemented by a neural network, while the white box is a pure symbolic logic evaluation. (A) The image is mapped to a concept prediction. (B) The image is mapped by the component selector to a distribution over rules. (C) This distribution is used to select a rule embedding from the encoded rulebook. (D) The rule embedding is decoded into a logic rule by assigning to each of the concepts its role in the rule, i.e. whether it is positive (P), negative (N), or irrelevant (I). Finally, (E) the rule is evaluated on the concept prediction to provide the task prediction on the task $\mathit{apple}$.
• Figure 3: Task accuracy on CelebA with varying numbers of employed concepts.
• Figure 4: Likelihoods to be maximized when $y=1$ w.r.t. the role $r$ of a single concept in a selected rule, for different situations. As $P+N+I=1$, irrelevance is the coordinate $(0,0)$. Likelihoods that cannot be achieved (i.e. when $P+N>1$) are put to 0. In the first column, the concept label is 1. In the second column, it is 0. In the third column, two examples with opposite labels select the rule.
• Figure 5: Rule interventions on CEBaB where we predict only the first task and employ 6 concepts. We compare CMR's accuracy with a CBM using a decision tree (DT) and CMR after adding the decision tree's rules to CMR's memory (DT $\rightarrow$ CMR) without any additional learnable rules (n = 0) and when allowing 15 additional learnable rules (n = 15). The mean and standard deviation is shown over 3 seeds. This means that (1) CMR's end-to-end rule learning allows it to obtain higher accuracy than when purely integrating pre-obtained rules from other rule learners (removing CMR's rule learning component), (2) just integrating pre-obtained rules in CMR (while still allowing more rules to be learned) does not decrease its accuracy, and (3) using CMR with only pre-obtained rules still surpasses the performance of the rules in isolation due to the selector.

Theorems & Definitions (5)

• Theorem 4.1
• proof
• Theorem 5.1: Log-likelihood
• proof
• proof