Discovering Dataset Nature through Algorithmic Clustering based on String Compression

TL;DR

This work tackles the problem of choosing a representation that preserves or discards text structure for clustering diverse text datasets. It employs Normalized Compression Distance (NCD) with three compressor types (LZMA, BZIP2, PPMD) and a distortion method that progressively removes frequent words while preserving structure to reveal dataset nature. The study shows that structural datasets benefit from structure-aware models (higher PPMD orders) whereas keyword-driven datasets benefit from structure-agnostic representations, and validates these findings with artificial grammar data and multidimensional projection comparisons. The approach provides a principled, compressor-based means to tailor text representations to dataset characteristics, with implications for more effective clustering and retrieval.

Abstract

Text datasets can be represented using models that do not preserve text structure, or using models that preserve text structure. Our hypothesis is that depending on the dataset nature, there can be advantages using a model that preserves text structure over one that does not, and viceversa. The key is to determine the best way of representing a particular dataset, based on the dataset itself. In this work, we propose to investigate this problem by combining text distortion and algorithmic clustering based on string compression. Specifically, a distortion technique previously developed by the authors is applied to destroy text structure progressively. Following this, a clustering algorithm based on string compression is used to analyze the effects of the distortion on the information contained in the texts. Several experiments are carried out on text datasets and artificially-generated datasets. The results show that in strongly structural datasets the clustering results worsen as text structure is progressively destroyed. Besides, they show that using a compressor which enables the choice of the size of the left-context symbols helps to determine the nature of the datasets. Finally, the results are contrasted with a method based on multidimensional projections and analogous conclusions are obtained.

Figures (9)

• Figure 1: Example of a dendrogram for the Books repository (a). Errorless dendrogram for the Books repository (b). Each leaf of the dendrogram corresponds to a document. The nodes labeled as "NM.TP" and "AP.AEoC" are incorrectly clustered in (a). This implies that the distances between the books by Niccolò Machiavelli (NM) and by Alexander Pope (AP) are higher than they should be if these nodes had been correctly clustered. The pairwise distances between the nodes belonging to this dendrogram can be seen in Table \ref{['Table.Dendro']}.
• Figure 2: Clustering results for the Books dataset and the PPMD (order 6) compression algorithm. Panels (a), (b) and (c) show the clustering error obtained for all the distortion techniques. Panel (d) shows the evolution of $1-DSC$ for all the distortion techniques. The clustering error gets worse as text structure is destroyed. This behavior is observed when both clustering error measures are used.
• Figure 3: Clustering results for the Books dataset and the BZIP2.
• Figure 4: Visual representation of the structural differences between the datasets. $DSC^r_{i}$ values for all the datasets and all the compressors using the distortion techniques RPA, RPRW and RPE. It can be observed that Books dataset and UCI-KDD dataset behave similarly, whereas IMDB dataset behaves like Medline dataset.
• Figure 5: PPMD order analysis. One can observe that depending on the nature of the dataset, the best clustering results are obtained using a PPMD order of 6 (UCI-KDD and Books) or a PPMD order of 2 (IMDB and Medline).