Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM

TL;DR

LLooM tackles the challenge of unstructured text analysis by replacing low-level keyword topics with high-level concepts defined by explicit inclusion criteria. The approach hinges on the LLooM algorithm, which uses large language models to synthesize concepts (Synthesize), and then score data against those concepts (Score), with auxiliary steps for data distillation and clustering to handle scale. The LLooM Workbench provides a mixed-initiative tool to visualize and interact with data through concepts, enabling theory-driven analysis and auditable traces back to lower-level evidence. Technical evaluations and expert case studies show that LLooM improves concept quality and data coverage compared to BERTopic, and can reveal novel insights in familiar datasets. The work discusses design implications, limitations of LLMs, and directions for making concept-driven analysis more reliable, transparent, and scalable.

Abstract

Data analysts have long sought to turn unstructured text data into meaningful concepts. Though common, topic modeling and clustering focus on lower-level keywords and require significant interpretative work. We introduce concept induction, a computational process that instead produces high-level concepts, defined by explicit inclusion criteria, from unstructured text. For a dataset of toxic online comments, where a state-of-the-art BERTopic model outputs "women, power, female," concept induction produces high-level concepts such as "Criticism of traditional gender roles" and "Dismissal of women's concerns." We present LLooM, a concept induction algorithm that leverages large language models to iteratively synthesize sampled text and propose human-interpretable concepts of increasing generality. We then instantiate LLooM in a mixed-initiative text analysis tool, enabling analysts to shift their attention from interpreting topics to engaging in theory-driven analysis. Through technical evaluations and four analysis scenarios ranging from literature review to content moderation, we find that LLooM's concepts improve upon the prior art of topic models in terms of quality and data coverage. In expert case studies, LLooM helped researchers to uncover new insights even from familiar datasets, for example by suggesting a previously unnoticed concept of attacks on out-party stances in a political social media dataset.

Figures (18)

• Figure 1: A process overview of the LLooM concept induction algorithm. Starting from (1) unstructured text data, LLooM performs (2) concept generation aided by an LLM to produce (3) high-level concepts, which consist of generated natural language descriptions and explicit criteria in the form of zero-shot LLM prompts. LLooM performs (4) concept scoring based on concept criteria prompts and visualizes data in terms of concepts in the (5) LLooM Workbench, a mixed-initiative text analysis tool.
• Figure 2: Concept generation in the LLooM algorithm, demonstrated with sample text inputs. The process starts with unstructured text data and an optional Seed from the analyst. Then, the Distill operator condenses the input data with an LLM by filtering to excerpts and summarizing to bullet points (gpt-3.5-turbo). The Cluster operator pools and groups the distilled bullet points using a clustering algorithm. Finally, the Synthesize operator proposes high-level concepts using an LLM prompt (gpt-4). The Loop operator can optionally repeat this process multiple times to produce higher-level concepts.
• Figure 3: The LLooM Workbench, an interactive text analysis tool that leverages LLooM's concept induction capabilities. The tool consists of the (A) Matrix view with an overview of the prevalence of concepts among user-defined data slices. Selecting a concept row displays the associated (B) Concept detail view, which displays the concept criteria, subconcepts, and matching examples. Selecting a slice column displays the corresponding (C) Slice detail view, which displays a similar overview of the examples within the slice.
• Figure 4: For the toxic content dataset, LLooM generates content-related concepts such as Empowerment of women and Gender inequality, but also surfaces style- and tone-related concepts such as Expressing frustration and Reflection and introspection.
• Figure 5: Scenario 1: Toxic content dataset---BERTopic places a large proportion of examples ($44.3\%$) in an uncategorized cluster (in grey) while most other clusters contain between $2-10\%$ of examples. LLooM concepts display a range of prevalence values from $1-50\%$, and the outlier category contains $9.5\%$ of examples.