Pre-Trained Models: Past, Present and Future

TL;DR

PTMs have transformed AI by enabling knowledge-rich representations learned from massive unlabeled data and fine-tuned for downstream tasks. The paper surveys the historical development from supervised transfer to self-supervised pre-training, and outlines architectural designs, data strategies, efficiency techniques, and interpretability theories driving current PTMs. It identifies open problems and future directions across architectures, multilingual/multimodal modalities, efficiency, theory, and applications. The overall contribution is a structured synthesis of past milestones and future opportunities for scalable, reliable PTMs.

Abstract

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

Figures (12)

• Figure 1: The two figures show the significant improvement on performance of both language understanding and language generation after using large-scale PTMs.
• Figure 2: Figure \ref{['fig:change3']} shows the number of publications with the keyword "language model" as well as their citations in different years. Figure \ref{['fig:change4']} shows the parameter size of large-scale PTMs for NLP tasks and the pre-training data size are increasing by $10$ times per year. From these figures, we can find that, after 2018, when large-scale NLP PTMs begin to be explored, more and more efforts are devoted to this field, and the model size and data size used by the PTMs are also getting larger.
• Figure 3: GPT-3, with 175 billion parameters, uses 560 GB data and 10,000 GPUs for its training. It has shown the abilities of learning world knowledge, common sense, and logical reasoning.
• Figure 4: The spectrum of pre-training methods from transfer learning, self-supervised learning to the latest pre-training neural models.
• Figure 5: The architecture of Transformer, GPT, and BERT.