Cost-constrained multi-label group feature selection using shadow features
Tomasz Klonecki, Paweł Teisseyre, Jaesung Lee
TL;DR
The paper tackles cost-constrained, multi-label feature selection where features are grouped and selecting a group provides access to all its features. It introduces a two-step shadow-feature method that avoids tuning a penalty parameter: first performing standard SFS with $\lambda=0$ to obtain $S_1$, then adding zero-cost features from already-used groups guided by a shadow-feature stopping rule based on $M_{\max}^* = \max M(X_k^*,Y|X_{S_1\cup S_2})$ versus the current candidate's $M(X_{k},Y|X_{S_1\cup S_2})$. To further simplify estimation, it derives low-dimensional MI-based score functions using a lower-bound approach with parameters $a,b$ (e.g., $a=b=1$), enabling efficient scoring without full conditional MI computation. Empirical evaluation on the MIMIC mimic2 dataset shows that the proposed method significantly improves performance over penalized SFS, especially at low budgets, demonstrating a practical, model-agnostic pathway to cost-effective medical prediction with grouped features.
Abstract
We consider the problem of feature selection in multi-label classification, considering the costs assigned to groups of features. In this task, the goal is to select a subset of features that will be useful for predicting the label vector, but at the same time, the cost associated with the selected features will not exceed the assumed budget. Solving the problem is of great importance in medicine, where we may be interested in predicting various diseases based on groups of features. The groups may be associated with parameters obtained from a certain diagnostic test, such as a blood test. Because diagnostic test costs can be very high, considering cost information when selecting relevant features becomes crucial to reducing the cost of making predictions. We focus on the feature selection method based on information theory. The proposed method consists of two steps. First, we select features sequentially while maximizing conditional mutual information until the budget is exhausted. In the second step, we select additional cost-free features, i.e., those coming from groups that have already been used in previous steps. Limiting the number of added features is possible using the stop rule based on the concept of so-called shadow features, which are randomized counterparts of the original ones. In contrast to existing approaches based on penalized criteria, in our method, we avoid the need for computationally demanding optimization of the penalty parameter. Experiments conducted on the MIMIC medical database show the effectiveness of the method, especially when the assumed budget is limited.