A Closer Look at Claim Decomposition

TL;DR

An LLM-based approach to generating decompositions inspired by Bertrand Russell’s theory of logical atomism and neo-Davidsonian semantics is proposed and improved decomposition quality is demonstrated over previous methods.

Abstract

As generated text becomes more commonplace, it is increasingly important to evaluate how well-supported such text is by external knowledge sources. Many approaches for evaluating textual support rely on some method for decomposing text into its individual subclaims which are scored against a trusted reference. We investigate how various methods of claim decomposition -- especially LLM-based methods -- affect the result of an evaluation approach such as the recently proposed FActScore, finding that it is sensitive to the decomposition method used. This sensitivity arises because such metrics attribute overall textual support to the model that generated the text even though error can also come from the metric's decomposition step. To measure decomposition quality, we introduce an adaptation of FActScore, which we call DecompScore. We then propose an LLM-based approach to generating decompositions inspired by Bertrand Russell's theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposition quality over previous methods.

Figures (7)

• Figure 1: Modes of claim decomposition. The extent to which textual support can be determined depends on how the generated text (yellow box) is decomposed into its subclaims (white boxes). Higher quality decompositions enable more complete identification of discrepancies between generated text and a reference (not shown), which consequently increases the reliablility of the downstream textual support metric. Checks and Xs denote that the statement is claimed or is not claimed, respectively, by the generated text.
• Figure 2: FActScore (macro-averaged across $\textrm{LM}_{\textrm{SUBJ}}$) using different decomposition methods. The same underlying set of documents is assigned different FActScore values depending on the decomposition method used.
• Figure 3: DecompScore (macro-averaged across $\textrm{LM}_{\textrm{SUBJ}}$) of different decomposition methods. A higher DecompScore is better.
• Figure 4: FActScore results for all claim decomposition methods and $\textrm{LM}_{\textrm{SUBJ}}$.
• Figure 5: FActScore results after filtering out subclaims determined to be not supported by the original sentence (using DecompScore judgments) for all claim decomposition methods and $\textrm{LM}_{\textrm{SUBJ}}$.