Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach
Jacob Carlson
TL;DR
The paper tackles open-ended discovery from unstructured data by converting observations $Z_i$ into high-dimensional dictionary features $Y_i \in \{0,1\}^p$ via a dictionary learning map $\texttt{Dict}$, then tests $p$ features with $T_n$ and $S_n$ to obtain interpretable discoveries. It develops new high-dimensional selective inference procedures that control the $k$-FWER under arbitrary dependence using Gaussian multiplier bootstrap and a small-$k$ central limit theory, enabling discoveries with strong familywise error guarantees even when $p \gg n$. The authors demonstrate the framework with two empirical economics applications, achieving additional, automatically interpretable insights beyond prior analyses, and provide open-source replication materials. The approach reduces researcher degrees of freedom, is cost-effective, and integrates autointerpretation pipelines to describe discovered concepts, offering a scalable path for principled, interpretable inference on unstructured data.
Abstract
Social scientists are increasingly turning to unstructured datasets to unlock new empirical insights, e.g., estimating causal effects on text outcomes, measuring beliefs from open-ended survey responses. In such settings, unsupervised analysis is often of interest, in that the researcher does not want to pre-specify the objects of measurement or otherwise artificially delimit the space of measurable concepts; they are interested in discovery. This paper proposes a general and flexible framework for pursuing discovery from unstructured data in a statistically principled way. The framework leverages recent methods from the literature on machine learning interpretability to map unstructured data points to high-dimensional, sparse, and interpretable dictionaries of concepts; computes (test) statistics of these dictionary entries; and then performs selective inference on them using newly developed statistical procedures for high-dimensional exceedance control of the $k$-FWER under arbitrary dependence. The proposed framework has few researcher degrees of freedom, is fully replicable, and is cheap to implement -- both in terms of financial cost and researcher time. Applications to recent descriptive and causal analyses of unstructured data in empirical economics are explored. An open source Jupyter notebook is provided for researchers to implement the framework in their own projects.