Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

Daniel Petrov

TL;DR

The paper questions the robustness of state-of-the-art NLI models by showing that high in-distribution accuracy can mask reliance on superficial cues. It uses Linguistically-Informed Transformations to generate a contrast set and demonstrates that exposing ELECTRA-small to a small amount of this data during fine-tuning yields large gains. Contrast-set accuracy rises from $74.9\%$ to $90.7\%$ (with SNLI performance around $89.3\%$), validating the approach. This work emphasizes the importance of diverse, challenging training data to achieve more semantically grounded language understanding and reliability in real-world scenarios.

Abstract

Large-scale pre-trained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research carried out in this study explores how the robustness of a language model can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model recovers performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.

From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

TL;DR

The paper questions the robustness of state-of-the-art NLI models by showing that high in-distribution accuracy can mask reliance on superficial cues. It uses Linguistically-Informed Transformations to generate a contrast set and demonstrates that exposing ELECTRA-small to a small amount of this data during fine-tuning yields large gains. Contrast-set accuracy rises from to (with SNLI performance around ), validating the approach. This work emphasizes the importance of diverse, challenging training data to achieve more semantically grounded language understanding and reliability in real-world scenarios.

Abstract

Large-scale pre-trained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research carried out in this study explores how the robustness of a language model can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model recovers performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.
Paper Structure (8 sections, 2 figures, 4 tables)

This paper contains 8 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Model & Dataset
  3. Model
  4. Training and Testing Data
  5. Contrast Set
  6. Method
  7. Model Improvement
  8. Conclusion

Figures (2)

  • Figure 1: Comparison between the original data and the perturbed data, showing how the model prediction accuracy decreases from simple changes to the premise and hypothesis.
  • Figure 2: Performance improvement of the model on contrast sets with varying training sample sizes.