Modular Deep Learning

TL;DR

Modular deep learning addresses transfer learning limitations by separating computation, routing, and aggregation into autonomous modules that can be combined and updated locally. The paper provides a unified taxonomy of computation functions (parameter, input, function, hypernetworks), routing strategies (fixed, learned hard/soft, with varying levels), and aggregation methods, and maps these to training settings and applications. It demonstrates that modularity enables positive transfer, continual learning, and parameter-efficient fine-tuning across NLP, speech, CV, and cross-modal tasks, with concrete case studies and design principles. The work highlights potential for scalable generalisation, domain knowledge injection, and multi-task to lifelong learning, and points to future directions such as modular instruction tuning and standardized evaluation.

Abstract

Transfer learning has recently become the dominant paradigm of machine learning. Pre-trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples. Nonetheless, it remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference and that generalise systematically to non-identically distributed tasks. Modular deep learning has emerged as a promising solution to these challenges. In this framework, units of computation are often implemented as autonomous parameter-efficient modules. Information is conditionally routed to a subset of modules and subsequently aggregated. These properties enable positive transfer and systematic generalisation by separating computation from routing and updating modules locally. We offer a survey of modular architectures, providing a unified view over several threads of research that evolved independently in the scientific literature. Moreover, we explore various additional purposes of modularity, including scaling language models, causal inference, programme induction, and planning in reinforcement learning. Finally, we report various concrete applications where modularity has been successfully deployed such as cross-lingual and cross-modal knowledge transfer. Related talks and projects to this survey, are available at https://www.modulardeeplearning.com/.

Figures (10)

• Figure 1: Case studies of modular deep learning; best viewed in colour. Green components illustrate different routing functions (see § \ref{['sec:routing']}); shade-of-purple components illustrate modular computation functions (see § \ref{['sec:nature_modularity']}). \ref{['fig:CaseStudy:MADX']}) MAD-X pfeiffer-etal-2020-mad uses Adapter layers with fixed routing for zero-shot cross-lingual transfer. \ref{['fig:CaseStudy:Polytropon']}) Polytropon ponti2022combining uses low-rank adapters hu2021lora with hard learned routing for few-shot task adaptation. \ref{['fig:CaseStudy:MoE']}) MoE Transformers fedus2021switchClark2022UnifiedScaling use Multi-Layer Perceptrons with top-$k$ soft routing, in order to scale to larger model sizes. The three representative models illustrated here are only a fraction of possible configurations from the 'configuration manifold' that can be created by varying the components surveyed in §\ref{['sec:nature_modularity']}-§\ref{['sec:training_setting']}.
• Figure 2: Different modular designs for Transformer architectures; best viewed in colour. Task-specific modular components are illustrated in magenta and purple, respectively. (a) Parameter Composition (§ \ref{['sec:nature_modularity:parameter_composition']}): A sparse sub-network in the linear layer as part of multi-head-attention. (b) Input Composition (§ \ref{['sec:nature_modularity:input_composition']}): Prefix-tuning Li2020PrefixTuning extends the input by prepending embeddings to the key and value matrices in the Transformer layer. (c) Function Composition (§ \ref{['sec:nature_modularity:layers']}): Task-specific bottleneck layers transforming the hidden representations are inserted in each layer houlsby2019parameter. (d) Hypernetwork (§ \ref{['sec:computation_function:hyper_network']}): A small separate neural network generates modular parameters conditioned on metadata. We show its application to function composition but it is compatible with all computation functions.
• Figure 3: Different approaches of function composition. (a) Sequential Bottleneck Adapter: The first adapter architecture proposed for transformers which consists of two bottleneck layers placed after the multi-head attention (MHA) and feed-forward (FF) layers houlsby2019parameter. (b) Parallel Bottleneck Adapter: Bottleneck layers processed in parallel to the MHA and FF layers of the pre-trained transformer components Rebuffi2018Adapters2Stickland2019BERTPALsHe2021UnifiedAdapters. (c) (IA)$^3$: Rescaling operations performed within the MHA and FF layers Liu2022IA3.
• Figure 4: Different routing methods. (a) Fixed Routing: Examples are passed to modules based on a pre-defined logic, known a priori. (b) Hard Learned Routing: Learned hard selection modules. (c) Soft Learned Routing: Soft selection and weighting of modules.
• Figure 5: Different routing levels. (a) Layer-wise Routing: The indices are chosen based on the input to the current layer. (b) Naive Global Routing: The same indices of modules are chosen for all the layers of the model. (c) Global Routing: The configuration (possibly different for each layer) is chosen globally.