Mixture of Concept Bottleneck Experts

TL;DR

Mixture of Concept Bottleneck Experts (M-CBEs) generalizes CBMs by enabling a mixture of experts and flexible, user-aligned functional forms for the concept-to-task mapping. The framework uses expression trees to represent each expert and aggregates predictions via a learned selector, with two instantiations: Linear M-CBE (parametric linear expressions) and Symbolic M-CBE (symbolic regression over user-defined operator vocabularies). Empirical results show that exploring this design space yields favorable accuracy-interpretability trade-offs, with Symbolic M-CBE achieving low-complexity, ground-truth-like mechanisms and strong intervenability, while Linear M-CBE scales well to high-dimensional concept spaces. Additionally, Symbolic M-CBE enables post-hoc adaptation by adjusting operator vocabularies, supporting diverse user needs without full retraining. Overall, M-CBEs offer a principled route to interpretable yet accurate concept-based predictions, adaptable to varying tasks and user constraints.

Abstract

Concept Bottleneck Models (CBMs) promote interpretability by grounding predictions in human-understandable concepts. However, existing CBMs typically fix their task predictor to a single linear or Boolean expression, limiting both predictive accuracy and adaptability to diverse user needs. We propose Mixture of Concept Bottleneck Experts (M-CBEs), a framework that generalizes existing CBMs along two dimensions: the number of experts and the functional form of each expert, exposing an underexplored region of the design space. We investigate this region by instantiating two novel models: Linear M-CBE, which learns a finite set of linear expressions, and Symbolic M-CBE, which leverages symbolic regression to discover expert functions from data under user-specified operator vocabularies. Empirical evaluation demonstrates that varying the mixture size and functional form provides a robust framework for navigating the accuracy-interpretability trade-off, adapting to different user and task needs.

Figures (8)

• Figure 1: Left: The plane defined by functional form and number of experts. Different CBMs occupy distinct positions in this space, with regions highlighted in different colors. In red we indicate our newly proposed instantiations. Right: The red curve denotes human constraints and the green curve denotes task constraints. The region bounded by the two curves represents the set of feasible models.
• Figure 2: Left: PGM representing the assumed generative process for M-CBEs. Right: Example of an expression tree ($T$) representing a linear expression. Green nodes are placeholder variables $V$, blue nodes are operators $O$ drawn from the vocabulary $\mathcal{W}$, and red nodes are learnable parameters $\Theta$. At inference time, each placeholder $v_i$ is instantiated with the corresponding predicted concept $c_i$.
• Figure 3: Overview of M-CBEs. The selector identifies the appropriate expression based on the input (e.g., parabolic motion). Simultaneously, the Concept Encoder predicts the underlying physical concepts ($x_0, v_0, \alpha, t$) from the raw input. The architecture accommodates different user mental models by allowing functional forms to be represented as either parametric linear equations or symbolic expressions restricted to interpretable operators. The final output $y$ is obtained by executing the selected function with the predicted concepts as arguments.
• Figure 4: The x-axis denotes model complexity, measured as the total number of nodes across all expression trees used by the task predictor. The y-axis reports MAE for regression tasks (top) and error rate for classification tasks (bottom). For multi-expert models, multiple points are shown corresponding to different numbers of experts (1–5). The dotted curve indicates the Pareto frontier, while the shaded region marks dominated solutions. Prior-M-CBE is excluded from the Pareto frontier as it relies on ground-truth expressions. CEM and BlackBox are plotted as horizontal lines, since their label-predictor complexity is undefined (they do not operate on concept predictions). Error bars indicate $95\%$ confidence intervals over five random seeds.
• Figure 5: Weight distributions for the concept “has underparts color brown” toward class "Crested Auklet", comparing LICEM (blue) and Lin-M-CBE (orange, memory size $= 2$).

Theorems & Definitions (4)

• Proposition 1
• Proposition 2
• proof
• proof