Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Prescriptive Deep Learning

Dimitris Bertsimas, Lisa Everest, Vasiliki Stoumpou

TL;DR

This work introduces Prescriptive Neural Networks (PNNs), a multimodal deep learning framework that integrates optimization with neural networks to output outcome-optimizing prescriptions from both structured and unstructured data. The method builds end-to-end embeddings, employs doubly robust counterfactual estimation, and uses a softmax-based policy learner to assign treatments, with interpretability recovered via Mirrored Optimal Classification Trees. Empirically, PNNs achieve significant outcome improvements on multimodal medical datasets (e.g., TAVR and liver trauma) and competitive results on unimodal tabular datasets, while maintaining stability and offering interpretable surrogates. The framework provides a flexible, scalable approach to data-driven prescription with realistic, robust prescriptions suitable for real-world deployment across diverse domains.

Abstract

We introduce a multimodal deep learning framework, Prescriptive Neural Networks (PNNs), that combines ideas from optimization and machine learning, and is, to the best of our knowledge, the first prescriptive method to handle multimodal data. The PNN is a feedforward neural network trained on embeddings to output an outcome-optimizing prescription. In two real-world multimodal datasets, we demonstrate that PNNs prescribe treatments that are able to significantly improve estimated outcomes in transcatheter aortic valve replacement (TAVR) procedures by reducing estimated postoperative complication rates by 32% and in liver trauma injuries by reducing estimated mortality rates by over 40%. In four real-world, unimodal tabular datasets, we demonstrate that PNNs outperform or perform comparably to other well-known, state-of-the-art prescriptive models; importantly, on tabular datasets, we also recover interpretability through knowledge distillation, fitting interpretable Optimal Classification Tree models onto the PNN prescriptions as classification targets, which is critical for many real-world applications. Finally, we demonstrate that our multimodal PNN models achieve stability across randomized data splits comparable to other prescriptive methods and produce realistic prescriptions across the different datasets.

Multimodal Prescriptive Deep Learning

TL;DR

This work introduces Prescriptive Neural Networks (PNNs), a multimodal deep learning framework that integrates optimization with neural networks to output outcome-optimizing prescriptions from both structured and unstructured data. The method builds end-to-end embeddings, employs doubly robust counterfactual estimation, and uses a softmax-based policy learner to assign treatments, with interpretability recovered via Mirrored Optimal Classification Trees. Empirically, PNNs achieve significant outcome improvements on multimodal medical datasets (e.g., TAVR and liver trauma) and competitive results on unimodal tabular datasets, while maintaining stability and offering interpretable surrogates. The framework provides a flexible, scalable approach to data-driven prescription with realistic, robust prescriptions suitable for real-world deployment across diverse domains.

Abstract

We introduce a multimodal deep learning framework, Prescriptive Neural Networks (PNNs), that combines ideas from optimization and machine learning, and is, to the best of our knowledge, the first prescriptive method to handle multimodal data. The PNN is a feedforward neural network trained on embeddings to output an outcome-optimizing prescription. In two real-world multimodal datasets, we demonstrate that PNNs prescribe treatments that are able to significantly improve estimated outcomes in transcatheter aortic valve replacement (TAVR) procedures by reducing estimated postoperative complication rates by 32% and in liver trauma injuries by reducing estimated mortality rates by over 40%. In four real-world, unimodal tabular datasets, we demonstrate that PNNs outperform or perform comparably to other well-known, state-of-the-art prescriptive models; importantly, on tabular datasets, we also recover interpretability through knowledge distillation, fitting interpretable Optimal Classification Tree models onto the PNN prescriptions as classification targets, which is critical for many real-world applications. Finally, we demonstrate that our multimodal PNN models achieve stability across randomized data splits comparable to other prescriptive methods and produce realistic prescriptions across the different datasets.
Paper Structure (28 sections, 7 equations, 2 figures, 10 tables)

This paper contains 28 sections, 7 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Contributions
  4. Methods
  5. Problem definition
  6. Embedding Extraction
  7. Structured data
  8. Unstructured data
  9. Counterfactual Estimation
  10. Counterfactual estimation of discrete treatments
  11. Counterfactual estimation of continuous treatments
  12. Prescription policy learning through feedforward neural networks
  13. Recovering Interpretability with Optimal Classification Trees
  14. Experiments with real-world datasets
  15. Data splits
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Architecture of a feedforward neural network (fnn-image).
  • Figure 2: Example of REBOA Mirrored OCT.