Who Should Have Surgery? A Comparative Study of GenAI vs Supervised ML for CRS Surgical Outcome Prediction

TL;DR

This study benchmarks calibrated tabular ML against multiple GenAI systems for pre-operative CRS outcome prediction using the SNOT-22 MCID threshold $8.9$ at 6 months. Across a merged multicenter dataset of 524 ESS cases, the in-house MLP outperforms GenAI models in discrimination (AUROC ≈ $0.66$) and calibration, making it the preferred primary predictor for triage decisions, while GenAI excels as an explainer aligned with clinician heuristics. Retrieval augmentation (RAG) did not improve predictive performance. The findings advocate an ML-first, GenAI-augmented workflow that leverages calibrated probabilities for decision-making and uses GenAI to enhance transparency, counseling, and shared decision-making. Prospective, multi-site validation and fairness monitoring are recommended for real-world deployment.

Abstract

Artificial intelligence has reshaped medical imaging, yet the use of AI on clinical data for prospective decision support remains limited. We study pre-operative prediction of clinically meaningful improvement in chronic rhinosinusitis (CRS), defining success as a more than 8.9-point reduction in SNOT-22 at 6 months (MCID). In a prospectively collected cohort where all patients underwent surgery, we ask whether models using only pre-operative clinical data could have identified those who would have poor outcomes, i.e. those who should have avoided surgery. We benchmark supervised ML (logistic regression, tree ensembles, and an in-house MLP) against generative AI (ChatGPT, Claude, Gemini, Perplexity), giving each the same structured inputs and constraining outputs to binary recommendations with confidence. Our best ML model (MLP) achieves 85 % accuracy with superior calibration and decision-curve net benefit. GenAI models underperform on discrimination and calibration across zero-shot setting. Notably, GenAI justifications align with clinician heuristics and the MLP's feature importance, repeatedly highlighting baseline SNOT-22, CT/endoscopy severity, polyp phenotype, and physchology/pain comorbidities. We provide a reproducible tabular-to-GenAI evaluation protocol and subgroup analyses. Findings support an ML-first, GenAI- augmented workflow: deploy calibrated ML for primary triage of surgical candidacy, with GenAI as an explainer to enhance transparency and shared decision-making.

Figures (4)

• Figure 1: Study pipeline. Pre-operative EHR variables are cleaned/encoded and used in two parallel paths: (top) calibrated supervised ML (LR/RF/MLP) and (bottom) tabular-to-LLM prompting for GenAI (ChatGPT/Claude/Gemini/Perplexity). Both output binary surgery recommendations (MCID $\geq$ 9 target).
• Figure 2: Confusion matrices (left of each pair) and ROC curves (right) for six models on the CRS test set ($n{=}105$). Rows (top$\to$bottom): ChatGPT 5 (Thinking), MedGPT– 5 (thinking), Gemini 2.5 Pro, Perplexity Sonar, Claude Sonnet 4.5, and our MLP. Numeric overlays show counts; ROC panels report the AUC for each model.
• Figure 3: Permutation feature importance for the MLP classifier on the held-out test set. Bars show the mean decrease in balanced accuracy when each feature is randomly permuted (larger values indicate greater importance); black ticks denote the standard deviation across permutation repeats. The model is most sensitive to SNOT22_BLN_TOTAL, Age, and BLN_CT_TOTAL, followed by ALLERGY_TESTING, PREVIOUS_SURGERY, and CRS_POLYPS.
• Figure 4: ChatGPT (GPT-5 Thinking) with RAG. Left: confusion matrix (TN$=3$, FP$=21$, FN$=1$, TP$=80$; accuracy $=0.79$). Right: ROC curve (AUC $=0.57$). RAG does not improve minority-class detection and leaves overall discrimination essentially unchanged.