A Survey of Deep Learning for Scientific Discovery

TL;DR

This survey maps the landscape of deep learning for scientific discovery, detailing models, tasks, and training paradigms across visuals, sequences, and graphs while emphasizing data efficiency and interpretability for scientific use. It presents a practical workflow and a taxonomy of methods, including self- and semi-supervised learning, transfer learning, and domain adaptation, complemented by extensive resources and tutorials to accelerate adoption. By synthesizing standard architectures with domain-specific considerations and providing implementation tips, the paper offers a concrete guide for scientists to select promising approaches and avoid common pitfalls. The work highlights community-driven assets, such as pretrained models and open-source tools, that enable rapid ramp-up and reproducible research in diverse scientific domains.

Abstract

Over the past few years, we have seen fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks. At the same time, the amount of data collected in a wide array of scientific domains is dramatically increasing in both size and complexity. Taken together, this suggests many exciting opportunities for deep learning applications in scientific settings. But a significant challenge to this is simply knowing where to start. The sheer breadth and diversity of different deep learning techniques makes it difficult to determine what scientific problems might be most amenable to these methods, or which specific combination of methods might offer the most promising first approach. In this survey, we focus on addressing this central issue, providing an overview of many widely used deep learning models, spanning visual, sequential and graph structured data, associated tasks and different training methods, along with techniques to use deep learning with less data and better interpret these complex models --- two central considerations for many scientific use cases. We also include overviews of the full design process, implementation tips, and links to a plethora of tutorials, research summaries and open-sourced deep learning pipelines and pretrained models, developed by the community. We hope that this survey will help accelerate the use of deep learning across different scientific domains.

Figures (14)

• Figure 1: Schematic of a typical deep learning workflow. A typical development process for deep learning applications can be viewed as consisting of three sequential stages (i) data related steps (ii) the learning component (iii) validation and analysis. Each one of these stages has several substeps and techniques associated with it, also depicted in the figure. In the survey we will overview most techniques in the learning component, as well as some techniques in the data and validation stages. Note that while a natural sequence is to first complete steps in the data stage, followed by learning and then validation, standard development will likely result in multiple different iterations where the techniques used or choices made in one stage are revisited based off of results of a later stage.
• Figure 2: The Supervised Learning process for training neural networks. The figure illustrates the supervised learning process for neural networks. Data instances (in this case images) and corresponding labels are collected. During the training step, the parameters of the neural network are optimized so that when input a data instance, the neural network outputs the corresponding label. During evaluation, the neural network is given unseen data instances as input, and if trained successfully, will output a meaningful label (prediction).
• Figure 3: Differences between Image Classification, Object Detection, Semantic Segmentation and Instance Segmentation tasks. Image source abdulla2018segmentation The figure illustrates the differences between classification, object detection, semantic segmentation and instance segmentation. In classification, the whole image gets a single label (balloons), while in object detection, each balloon is also localized with a bounding box. In semantic segmentation, all the pixels corresponding to balloon are identified, while in instance segmentation, each individual balloon is identified separately.
• Figure 4: Pose Estimation. Image source sun2019deep The task of pose estimation, specifically multi-person 2D (human) pose-estimation is depicted in the figure. The neural network model predicts the positions of the main joints (keypoints), which are combined with a body model to get the stick-figure like approximations of pose overlaid on the multiple humans in the image. Variants of these techniques have been used to study animal behaviors in scientific settings.
• Figure 5: Illustration of the Sequence to Sequence prediction task. Image source zhang2019dive The figure shows an illustration of a Sequence to Sequence task, translating an input sentence (sequence of tokens) in English to an output sentence in German. Note the encoder-decoder structure of the underlying neural network, with the encoder taking in the input, and the decoder generating the output, informed by the encoder representations and the previously generated output tokens. In this figure, the input tokens are fed in one by one, and the output is also generated one at a time, which is the paradigm when using Recurrent Neural Networks as the underlying model. With Transformer models, which are now extremely popular for sequence to sequence tasks, the sequence is input all at once, significantly speeding up use.