Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text

TL;DR

This paper addresses the limitations of random, fixed-rate masking in masked language modeling during fine-tuning by introducing Task-Informed Anti-Curriculum by Masking (TIACBM). TIACBM uses a cyclic decaying masking ratio together with task-specific token relevance to selectively mask impactful tokens for downstream tasks, and it demonstrates consistent, statistically significant gains across sentiment analysis, topic classification, and authorship attribution on multiple model families. The method shows robustness across datasets and tasks, and its design highlights the value of incorporating downstream-task priors into the masking process. The work also provides a general, code-released approach applicable to various LLMs, suggesting practical improvements for downstream performance beyond pre-training.

Abstract

Masked language modeling has become a widely adopted unsupervised technique to pre-train large language models (LLMs). However, the process of selecting tokens for masking is random, and the percentage of masked tokens is typically fixed for the entire training process. In this paper, we propose to adjust the masking ratio and to decide which tokens to mask based on a novel task-informed anti-curriculum learning scheme. First, we harness task-specific knowledge about useful and harmful tokens in order to determine which tokens to mask. Second, we propose a cyclic decaying masking ratio, which corresponds to an anti-curriculum schedule (from hard to easy). We exemplify our novel task-informed anti-curriculum by masking (TIACBM) approach across three diverse downstream tasks: sentiment analysis, text classification by topic, and authorship attribution. Our findings suggest that TIACBM enhances the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. We release our code at https://github.com/JarcaAndrei/TIACBM.