Word Matters: What Influences Domain Adaptation in Summarization?
Yinghao Li, Siyu Miao, Heyan Huang, Yang Gao
TL;DR
This work examines how word-level properties in training data influence domain adaptation for abstractive summarization. It introduces two learning-difficulty indicators, Compression Ratio $\alpha$ and Abstraction Level $\beta$, and combines them into a learning difficulty coefficient $\lambda = \alpha \beta$, to explain transfer performance. The authors show that cross-domain overlap $\gamma$ linearly relates to performance gains when accounting for $\lambda$ (LD-Gain), while total word count has no consistent effect. They further demonstrate a predictive framework where unseen-domain performance can be estimated from $\lambda$ and $\gamma$ without retraining, offering a resource-efficient way to anticipate domain transfer for new domains across multiple models and datasets. The findings suggest prioritizing word-level domain similarity over data volume and enable preemptive domain suitability assessment for LLM summarization systems.
Abstract
Domain adaptation aims to enable Large Language Models (LLMs) to generalize domain datasets unseen effectively during the training phase. However, factors such as the size of the model parameters and the scale of training data are general influencers and do not reflect the nuances of domain adaptation performance. This paper investigates the fine-grained factors affecting domain adaptation performance, analyzing the specific impact of `words' in training data on summarization tasks. We propose quantifying dataset learning difficulty as the learning difficulty of generative summarization, which is determined by two indicators: word-based compression rate and abstraction level. Our experiments conclude that, when considering dataset learning difficulty, the cross-domain overlap and the performance gain in summarization tasks exhibit an approximate linear relationship, which is not directly related to the number of words. Based on this finding, predicting a model's performance on unknown domain datasets is possible without undergoing training.