Visual Analytics for Fine-grained Text Classification Models and Datasets

TL;DR

SemLa is a novel Visual Analytics system tailored for dissecting complex semantic structures in a dataset when it is spatialized in model embedding space and visualizing fine‐grained nuances in the meaning of text samples to faithfully explain model reasoning.

Abstract

In natural language processing (NLP), text classification tasks are increasingly fine-grained, as datasets are fragmented into a larger number of classes that are more difficult to differentiate from one another. As a consequence, the semantic structures of datasets have become more complex, and model decisions more difficult to explain. Existing tools, suited for coarse-grained classification, falter under these additional challenges. In response to this gap, we worked closely with NLP domain experts in an iterative design-and-evaluation process to characterize and tackle the growing requirements in their workflow of developing fine-grained text classification models. The result of this collaboration is the development of SemLa, a novel visual analytics system tailored for 1) dissecting complex semantic structures in a dataset when it is spatialized in model embedding space, and 2) visualizing fine-grained nuances in the meaning of text samples to faithfully explain model reasoning. This paper details the iterative design study and the resulting innovations featured in SemLa. The final design allows contrastive analysis at different levels by unearthing lexical and conceptual patterns including biases and artifacts in data. Expert feedback on our final design and case studies confirm that SemLa is a useful tool for supporting model validation and debugging as well as data annotation.

Figures (3)

• Figure 2: Timeline of the iterative design process.
• Figure 3: Multi-level analysis results: a) local words in the two top-confused labels getting_spare_card and top_up_by_card_charge, b) label topping_up_by_card_charge has the word "card" more often than the label getting_spare_card in the model predictions, as the word "card" appears over the former but not the latter at frequency threshold of 30, and c) token-to-token links in a false positive case of getting_spare_card (represented by an example on the left) being mistaken for topping_up_by_card_charge (represented by an example on the right) confirms that the word "card" was a confounding feature.
• Figure 4: The concept of "a country" connecting the label vaccine to cancel_reservation