Finding Convincing Views to Endorse a Claim

TL;DR

The paper tackles the risk of cherry-picked data-based claims by reframing evaluation as claim endorsement: given a claim, it seeks natural subpopulations (views) that make the claim hold. It introduces an anytime framework that first ranks attribute-combinations and then exhaustively searches value assignments to produce refinements, guided by multiple naturalness measures such as Embedding similarity, ANOVA, MI, and coverage. Empirical results across ACS, Stack Overflow, and Flights show the approach yields high-quality refinements quickly, with Merged Top-k outperforming baselines in both speed and recall; a user study confirms the naturalness measures align with human intuition. The work also provides extensive case studies and ablations, demonstrating the utility of the framework for critical data analysis and highlighting avenues for future extensions to richer predicate spaces and interactive workflows.

Abstract

Recent studies investigated the challenge of assessing the strength of a given claim extracted from a dataset, particularly the claim's potential of being misleading and cherry-picked. We focus on claims that compare answers to an aggregate query posed on a view that selects tuples. The strength of a claim amounts to the question of how likely it is that the view is carefully chosen to support the claim, whereas less careful choices would lead to contradictory claims. We embark on the study of the reverse task that offers a complementary angle in the critical assessment of data-based claims: given a claim, find useful supporting views. The goal of this task is twofold. On the one hand, we aim to assist users in finding significant evidence of phenomena of interest. On the other hand, we wish to provide them with machinery to criticize or counter given claims by extracting evidence of opposing statements. To be effective, the supporting sub-population should be significant and defined by a ``natural'' view. We discuss several measures of naturalness and propose ways of extracting the best views under each measure (and combinations thereof). The main challenge is the computational cost, as naïve search is infeasible. We devise anytime algorithms that deploy two main steps: (1) a preliminary construction of a ranked list of attribute combinations that are assessed using fast-to-compute features, and (2) an efficient search for the actual views based on each attribute combination. We present a thorough experimental study that shows the effectiveness of our algorithms in terms of quality and execution cost. We also present a user study to assess the usefulness of the naturalness measures.

Figures (4)

• Figure 1: Top 100 score recall for each naturalness measure over time for our methods and external baselines for ACS.
• Figure 2: User study results: average rating and times ranked top for each method. (gf) stands for generality filter.
• Figure 3: Time for 95% score recall of avg. naturalness over ACS with at most $m=2$ atoms, with varying number of (a) tuples, (b) columns, and (c) varying values of $k$. (d) Sensitivity to maximal number of atoms ($m$): time for 95% recall of average naturalness with increasing number of atoms over Stack Overflow with 10 attributes.
• Figure 4: Top 100 score recall for each naturalness measure over time for various sampling sizes in sampling guided search over the ACS dataset.

Theorems & Definitions (11)

• Example 1.1
• Example 1.2
• Example 2.1
• Definition 2.2: Claim
• Example 2.3
• Definition 2.4: Refinement
• Example 2.5
• Definition 2.6: Naturalness Measure
• Definition 2.7: Claim Endorsement
• Example 2.8