Can Similarity-Based Domain-Ordering Reduce Catastrophic Forgetting for Intent Recognition?

TL;DR

This study examines whether arranging training domains by similarity can mitigate catastrophic forgetting in continual learning for intent recognition. It compares three domain-ordering strategies (min-sum path, max-sum path, random) using T5-based models on the SGD corpus, evaluating with Average Accuracy and Average CF. The key finding is that min-sum path reduces forgetting for the smaller T5-Base model, but this advantage vanishes for the larger T5-Large model, indicating model capacity influences the usefulness of domain ordering. The work suggests domain ordering as a viable supplementary technique in resource-constrained settings, with curriculum-like progression offering stability gains for smaller models.

Abstract

Task-oriented dialogue systems are expected to handle a constantly expanding set of intents and domains even after they have been deployed to support more and more functionalities. To live up to this expectation, it becomes critical to mitigate the catastrophic forgetting problem (CF) that occurs in continual learning (CL) settings for a task such as intent recognition. While existing dialogue systems research has explored replay-based and regularization-based methods to this end, the effect of domain ordering on the CL performance of intent recognition models remains unexplored. If understood well, domain ordering has the potential to be an orthogonal technique that can be leveraged alongside existing techniques such as experience replay. Our work fills this gap by comparing the impact of three domain-ordering strategies (min-sum path, max-sum path, random) on the CL performance of a generative intent recognition model. Our findings reveal that the min-sum path strategy outperforms the others in reducing catastrophic forgetting when training on the 220M T5-Base model. However, this advantage diminishes with the larger 770M T5-Large model. These results underscores the potential of domain ordering as a complementary strategy for mitigating catastrophic forgetting in continually learning intent recognition models, particularly in resource-constrained scenarios.

Figures (3)

• Figure 1: In continual learning, the model is trained one domain at the time. In this example, the model is first trained on data from the event domain $D_{event}$, then trained based on the payment domain $D_{payment}$ and so on. "request payment" might be mislabeled as "play payment" after $D_{media}$ training due to CF.
• Figure 2: Random, Min-Sum Path, and Max-Sum Path ordering strategies.
• Figure 3: Density plots of Average Accuracy and Average CF (Forgetting) for min-sum path, max-sum path and random ordering strategies using T5-Base and T5-Large models across 22 domain subsets. Min-sum-path strategy with T5-Base results in minimal Average CF, while no significant differences are observed in Average Accuracy between strategies or with T5-Large models.