LLM Augmentations to support Analytical Reasoning over Multiple Documents

TL;DR

This work explores if and how LLMs can be helpful to analysts for this task and develops an architecture to augment the capabilities of an LLM with a memory module called dynamic evidence trees (DETs) to develop and track multiple investigation threads.

Abstract

Building on their demonstrated ability to perform a variety of tasks, we investigate the application of large language models (LLMs) to enhance in-depth analytical reasoning within the context of intelligence analysis. Intelligence analysts typically work with massive dossiers to draw connections between seemingly unrelated entities, and uncover adversaries' plans and motives. We explore if and how LLMs can be helpful to analysts for this task and develop an architecture to augment the capabilities of an LLM with a memory module called dynamic evidence trees (DETs) to develop and track multiple investigation threads. Through extensive experiments on multiple datasets, we highlight how LLMs, as-is, are still inadequate to support intelligence analysts and offer recommendations to improve LLMs for such intricate reasoning applications.

Figures (11)

• Figure 1: Three steps to intelligence analysis (IA).
• Figure 2: Sample intelligence report
• Figure 3: Dynamic Evidence Trees
• Figure 4: Data condensation and dot extraction.
• Figure 5: All augmentations together with retrieve and merge