Large Causal Models from Large Language Models

TL;DR

DEMOCRITUS introduces a language-enabled framework to build large causal models by harvesting thousands of causal claims from LLMs and weaving them into domain-specific topos slices called large causal models. The core pipeline combines topic discovery, causal question/statement generation, relational triple extraction, and geometric refinement via a Geometric Transformer, producing navigable 2D/3D manifolds and local neighborhoods. Across economics, biology, and archaeology, the approach yields coherent macro-structures, hub-driven causal hubs, and cross-domain bridges, while highlighting limitations such as reliance on LLM content, lack of identifiability, and high compute costs that motivate active manifold building. The work lays a foundation for future dynamical and mechanistic models by linking textual causal narratives to structured, verifiable causal tools and simulations, with potential extensions to ODE-based or agent-based representations. The DEMOCRITUS framework thus offers a scalable, interpretable, and domain-spanning tool for hypothesis generation and exploratory causal reasoning, pending further validation and integration with quantitative causal inference methods.

Abstract

We introduce a new paradigm for building large causal models (LCMs) that exploits the enormous potential latent in today's large language models (LLMs). We describe our ongoing experiments with an implemented system called DEMOCRITUS (Decentralized Extraction of Manifold Ontologies of Causal Relations Integrating Topos Universal Slices) aimed at building, organizing, and visualizing LCMs that span disparate domains extracted from carefully targeted textual queries to LLMs. DEMOCRITUS is methodologically distinct from traditional narrow domain and hypothesis centered causal inference that builds causal models from experiments that produce numerical data. A high-quality LLM is used to propose topics, generate causal questions, and extract plausible causal statements from a diverse range of domains. The technical challenge is then to take these isolated, fragmented, potentially ambiguous and possibly conflicting causal claims, and weave them into a coherent whole, converting them into relational causal triples and embedding them into a LCM. Addressing this technical challenge required inventing new categorical machine learning methods, which we can only briefly summarize in this paper, as it is focused more on the systems side of building DEMOCRITUS. We describe the implementation pipeline for DEMOCRITUS comprising of six modules, examine its computational cost profile to determine where the current bottlenecks in scaling the system to larger models. We describe the results of using DEMOCRITUS over a wide range of domains, spanning archaeology, biology, climate change, economics, medicine and technology. We discuss the limitations of the current DEMOCRITUS system, and outline directions for extending its capabilities.

Figures (14)

• Figure 1: A local neighborhood causal model from a large LCM learned by DEMOCRITUS concerning the Indus river and droughts indus_valley_collapse, which was used to explain the collapse of the Indus Valley Civilization 5000 years ago. Crucially, such a model cannot be learned by a single prompt to an LLM, but instead is woven together by an assembly of carefully curated prompts.
• Figure 2: A global LCM underlying the collapse of the ancient Indus Valley civilization, shown as a 2D UMAP manifold of nodes colored by domain. Clusters correspond to Harappan urbanism and water systems, Indus script and epigraphy, climate and hydrology (Holocene monsoon variability, Indus river discharge and droughts), agriculture, and trade connections with Egypt and Rome, among others.
• Figure 3: A neighborhood region of an LCM around the topic "Building Long-Term Influencer Partnerships vs. One-Time Campaigns". Nodes are variables/phrases; red arrows indicate causal directions; node size/colour reflects GT activations. This structure arises from DEMOCRITUS' multi-step pipeline (topic graph, causal statements, triple extraction, GT), not from a single prompt.
• Figure 4: Democritus LCM. 3D UMAP projection of an LCM constructed from over $90,000$ causal textual statements sampled from GPT models in over $10$ domains. The structure exhibits clear domain clustering and smooth causal transition regions.
• Figure 5: A 2D UMAP projection of an LCM for biology.