ROOFS: RObust biOmarker Feature Selection

TL;DR

roofs introduces a comprehensive, stability-focused framework for benchmarking feature selection methods on biomedical data, addressing dataset- and task-dependent variability that often hampers biomarker discovery. By employing bootstrap resampling, optimism-corrected performance, and semi-synthetic truth, roofs enables data-driven selection of FS methods tailored to a given predictive goal. Application to the PIONeeR NSCLC dataset shows that simple BH-adjusted p-value filters can outperform more complex methods in stability and predictive performance, while still achieving high true-positive rates when needed. The framework thus promises to improve the robustness and translational value of clinical predictive models by guiding method choice and providing transparent performance reporting.

Abstract

Feature selection (FS) is essential for biomarker discovery and in the analysis of biomedical datasets. However, challenges such as high-dimensional feature space, low sample size, multicollinearity, and missing values make FS non-trivial. Moreover, FS performances vary across datasets and predictive tasks. We propose roofs, a Python package available at https://gitlab.inria.fr/compo/roofs, designed to help researchers in the choice of FS method adapted to their problem. Roofs benchmarks multiple FS methods on the user's data and generates reports that summarize a comprehensive set of evaluation metrics, including downstream predictive performance estimated using optimism correction, stability, reliability of individual features, and true positive and false positive rates assessed on semi-synthetic data with a simulated outcome. We demonstrate the utility of roofs on data from the PIONeeR clinical trial, aimed at identifying predictors of resistance to anti-PD-(L)1 immunotherapy in lung cancer. The PIONeeR dataset contained 374 multi-source blood and tumor biomarkers from 435 patients. A reduced subset of 214 features was obtained through iterative variance inflation factor pre-filtering. Of the 34 FS methods gathered in roofs, we evaluated 23 in combination with 11 classifiers (253 models in total) and identified a filter based on the union of Benjamini-Hochberg false discovery rate-adjusted p-values from t-test and logistic regression as the optimal approach, outperforming other methods including the widely used LASSO. We conclude that comprehensive benchmarking with roofs has the potential to improve the robustness and reproducibility of FS discoveries and increase the translational value of clinical models.

Figures (6)

• Figure 1: Overview of the roofs pipeline for comprehensive FS benchmarking.
• Figure 2: Instability of LASSO. A: Bootstrap selection frequencies for features selected by LASSO in at least 1 bootstrap sample. B: Variability in selected subset sizes and out-of-bag AUC.
• Figure 3: Improvement in FS stability following multicollinearity reduction with VIF pre-filtering compared to the original dataset. A: Correlation structure of the dataset before and after applying VIF pre-filtering. B: Absolute change in FS stability across representative FS methods.
• Figure 4: Performance of representative FS methods from different algorithmic families (top: FS methods with fixed, user-defined subset size; bottom: FS methods with subset size based on algorithm thresholds). Blue: embedded methods; violet-blue: resampling-based ensemble methods; green: filters; rose: wrappers. A, C: Trade-off between discovery and error in the experiment on the semi-synthetic PIONeeR dataset with a simulated outcome. The true positive rate denotes the proportion of true predictors correctly identified by each FS method, while the false positive rate shows the proportion of non-informative features incorrectly selected. Error bars represent one-half of the standard deviation (SD). B, D: Stability and predictive performance (AUC) in the experiment on the real PIONeeR dataset.
• Figure 5: Comparison of performance between Shapicant (representative wrapper method), LASSO (baseline embedded method), and p.adjust (the FS approach selected using roofs). A: Selection frequency of signature features. B: Instability index, defined as the proportion of bootstrap models yielding a different classification for a given patient compared to the full model Riley_Collins_2023. C: Out-of-bag AUC ($\theta_{\text{OOB}}$; see Methods) as a function of selected subset size across bootstrap models.