Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?

Jakub Hościłowicz, Marcin Sowański, Piotr Czubowski, Artur Janicki

TL;DR

It is concluded that quantification of information decodability is critical for many practical applications of the probing paradigm and consequently to build better NLU models.

Abstract

In this article, we use probing to investigate phenomena that occur during fine-tuning and knowledge distillation of a BERT-based natural language understanding (NLU) model. Our ultimate purpose was to use probing to better understand practical production problems and consequently to build better NLU models. We designed experiments to see how fine-tuning changes the linguistic capabilities of BERT, what the optimal size of the fine-tuning dataset is, and what amount of information is contained in a distilled NLU based on a tiny Transformer. The results of the experiments show that the probing paradigm in its current form is not well suited to answer such questions. Structural, Edge and Conditional probes do not take into account how easy it is to decode probed information. Consequently, we conclude that quantification of information decodability is critical for many practical applications of the probing paradigm.

Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?

TL;DR

It is concluded that quantification of information decodability is critical for many practical applications of the probing paradigm and consequently to build better NLU models.

Abstract

In this article, we use probing to investigate phenomena that occur during fine-tuning and knowledge distillation of a BERT-based natural language understanding (NLU) model. Our ultimate purpose was to use probing to better understand practical production problems and consequently to build better NLU models. We designed experiments to see how fine-tuning changes the linguistic capabilities of BERT, what the optimal size of the fine-tuning dataset is, and what amount of information is contained in a distilled NLU based on a tiny Transformer. The results of the experiments show that the probing paradigm in its current form is not well suited to answer such questions. Structural, Edge and Conditional probes do not take into account how easy it is to decode probed information. Consequently, we conclude that quantification of information decodability is critical for many practical applications of the probing paradigm.
Paper Structure (12 sections, 1 equation, 3 figures, 2 tables)

This paper contains 12 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. METHODS AND MODELS
  4. Probing Methods Used
  5. Joint NLU
  6. Knowledge Distillation for Joint NLU
  7. Data
  8. EXPERIMENTAL RESULTS
  9. Fine-tuning Analysis
  10. Probing Knowledge Distillation
  11. DISCUSSION
  12. CONCLUSIONS AND FUTURE WORK

Figures (3)

  • Figure 1: Probing interpretation in typical NLU scenario. The "Architecture" sub-graph describes modeling design and terminology. "Environment" gives an overview of the pipeline. Finally, the "Probing Interpretation" sub-graph describes how probing can be interpreted and what limitations we see.
  • Figure 2: Influence of dataset size on NLU accuracy and probing results.
  • Figure 3: Structural and edge probing results in fine-tuning scenario on Leyzer and UD datasets, for three variants of NLU architecture.