Predicting Practically? Domain Generalization for Predictive Analytics in Real-world Environments

TL;DR

The paper tackles the challenge of distribution shifts in real-world CRM predictive analytics by introducing GRADFrame, a Domain Generalization method based on Distributionally Robust Optimization. GRADFrame builds a hypothetical distribution space per source domain that enforces covariate and concept shift constraints and uses a two-stage min-max optimization to generate worst-case fictitious data for robust training. Through simulations and a real customer churn dataset, the authors demonstrate that GRADFrame outperforms a wide range of baselines, particularly under temporal and spatial shifts, while also highlighting the limitations of purely domain-invariant approaches. The work contributes a principled data-augmentation artifact, practical hyperparameter tuning via Leave-One-Domain-Out cross-validation, and design insights for robust predictive analytics in evolving business environments, with implications for Information Systems design and AI resilience.

Abstract

Predictive machine learning models are widely used in customer relationship management (CRM) to forecast customer behaviors and support decision-making. However, the dynamic nature of customer behaviors often results in significant distribution shifts between training data and serving data, leading to performance degradation in predictive models. Domain generalization, which aims to train models that can generalize to unseen environments without prior knowledge of their distributions, has become a critical area of research. In this work, we propose a novel domain generalization method tailored to handle complex distribution shifts, encompassing both covariate and concept shifts. Our method builds upon the Distributionally Robust Optimization framework, optimizing model performance over a set of hypothetical worst-case distributions rather than relying solely on the training data. Through simulation experiments, we demonstrate the working mechanism of the proposed method. We also conduct experiments on a real-world customer churn dataset, and validate its effectiveness in both temporal and spatial generalization settings. Finally, we discuss the broader implications of our method for advancing Information Systems (IS) design research, particularly in building robust predictive models for dynamic managerial environments.

Figures (8)

• Figure 1: Two scenarios of customer data distribution shift: covariate shift (Left) vs. concept shift (Right).
• Figure 2: Simulation Visualizations.
• Figure 3: Distribution of the extent of covariate shift across different $\gamma_1$ values (left). Distribution of the extent of concept shift across different $\gamma_2$ values.
• Figure 4: Evidence of customer distribution shift in temporal generalization scenario (left) and spatial generalization scenario (right) .
• Figure 5: SHAP value distributions for the top 10 features in churn prediction models trained on source domain (left) and target domain (right).