Interaction Concordance Index: Performance Evaluation for Interaction Prediction Methods

TL;DR

This work introduces the IC-index, a targeted performance estimator that measures how well predictive models capture the directions of drug–target interaction effects beyond additive main effects. By formalizing a distribution-level and sample-based framework, the IC-index complements traditional metrics like C-index and AUC, and its invariances to the grand mean and main effects ensure it isolates interaction behavior. The authors analyze permutation equivariance and the necessity of side information to generalize interaction predictions to unseen drugs or targets, provide efficient computation strategies, and validate the approach across seven biomedical datasets and multiple learning algorithms. Their empirical results show that IC-index can differentiate models that appear similar under conventional metrics, underscoring its practical value for guiding model development and data collection in interaction prediction. An open-source implementation is provided for computing IC-index efficiently in large-scale settings.

Abstract

Consider two sets of entities and their members' mutual affinity values, say drug-target affinities (DTA). Drugs and targets are said to interact in their effects on DTAs if drug's effect on it depends on the target. Presence of interaction implies that assigning a drug to a target and another drug to another target does not provide the same aggregate DTA as the reversed assignment would provide. Accordingly, correctly capturing interactions enables better decision-making, for example, in allocation of limited numbers of drug doses to their best matching targets. Learning to predict DTAs is popularly done from either solely from known DTAs or together with side information on the entities, such as chemical structures of drugs and targets. In this paper, we introduce interaction directions' prediction performance estimator we call interaction concordance index (IC-index), for both fixed predictors and machine learning algorithms aimed for inferring them. IC-index complements the popularly used DTA prediction performance estimators by evaluating the ratio of correctly predicted directions of interaction effects in data. First, we show the invariance of IC-index on predictors unable to capture interactions. Secondly, we show that learning algorithm's permutation equivariance regarding drug and target identities implies its inability to capture interactions when either drug, target or both are unseen during training. In practical applications, this equivariance is remedied via incorporation of appropriate side information on drugs and targets. We make a comprehensive empirical evaluation over several biomedical interaction data sets with various state-of-the-art machine learning algorithms. The experiments demonstrate how different types of affinity strength prediction methods perform in terms of IC-index complementing existing prediction performance estimators.

Figures (11)

• Figure 1: The figure illustrates the partition of drug-target pairs based on their components presence in the training data. The blue colored squares represent the in-training-set drug-target pairs having observed affinity values in the training data. The green area represent the off-training-set pair composed of in-training-set drugs and targets. The two yellow colored blocks represent pairs composed of either in-training-set drugs and off-training-set targets or vice versa. The red colored pairs are composed of off-training-set drugs and targets. The grey-colored blocks represent side information associated with either drugs or targets. For example, the row marked with $\phi(d_i)$ can be a feature representation associated to the drug $d_i$.
• Figure 2: An example of training-test splits corresponding to different OTS problems: in all problems, black squares denote the training set. The test set for the IDIT problem may consist of any blue datum, while test sets IDOT and ODIT problems may contain any dark yellow and light yellow datum, respectively. Finally, any red datum may be included in the test set simulating the ODOT problem.
• Figure 3: A visual example of imbalanced XOR data.
• Figure 4: Average prediction performances on test data and symmetric 95 % credible intervals over $10^6$ repetitions using imbalanced XOR data for the six considered learning algorithms.
• Figure 5: Results for the Davis data set.

Theorems & Definitions (32)

• Example 1
• Remark 1
• Definition 1: Partition of off-training-set drug-target pairs
• Definition 2: Learning algorithm
• Definition 3: Utility function
• Remark 2: Default utility value 0.5
• Definition 4: Binary classification utility
• Definition 5: Concordance
• Definition 6: Drugwise and targetwise concordance
• Definition 7: Estimand