A statistical significance testing approach for measuring term burstiness with applications to domain-specific terminology extraction
Samuel Sarria Hurtado, Todd Mullen, Taku Onodera, Paul Sheridan
TL;DR
This work tackles the problem of measuring term burstiness by moving beyond traditional chi-squared tests to a significance-testing framework. It derives a theoretical link between IDF and ICF under a multinomial language model and introduces RICF, a computationally efficient surrogate that blends both statistics to score burstiness. Through GENIA-based terminology extraction experiments and stopword analyses, the approach proves competitive with established burstiness measures and can outperform them in practical filtering tasks, while KeyBERT often achieves the best overall performance. The results suggest meaningful potential for incorporating significance testing into domain-specific text analysis and point to future enhancements via more exact tail probability approximations and embedding-assisted hybrids.
Abstract
A term in a corpus is said to be ``bursty'' (or overdispersed) when its occurrences are concentrated in few out of many documents. In this paper, we propose Residual Inverse Collection Frequency (RICF), a statistical significance test inspired heuristic for quantifying term burstiness. The chi-squared test is, to our knowledge, the sole test of statistical significance among existing term burstiness measures. Chi-squared test term burstiness scores are computed from the collection frequency statistic (i.e., the proportion that a specified term constitutes in relation to all terms within a corpus). However, the document frequency of a term (i.e., the proportion of documents within a corpus in which a specific term occurs) is exploited by certain other widely used term burstiness measures. RICF addresses this shortcoming of the chi-squared test by virtue of its term burstiness scores systematically incorporating both the collection frequency and document frequency statistics. We evaluate the RICF measure on a domain-specific technical terminology extraction task using the GENIA Term corpus benchmark, which comprises 2,000 annotated biomedical article abstracts. RICF generally outperformed the chi-squared test in terms of precision at k score with percent improvements of 0.00% (P@10), 6.38% (P@50), 6.38% (P@100), 2.27% (P@500), 2.61% (P@1000), and 1.90% (P@5000). Furthermore, RICF performance was competitive with the performances of other well-established measures of term burstiness. Based on these findings, we consider our contributions in this paper as a promising starting point for future exploration in leveraging statistical significance testing in text analysis.