Encoding and Understanding Astrophysical Information in Large Language Model-Generated Summaries

TL;DR

The paper asks whether LLM-generated summaries can encode physical information derived from astrophysical measurements and how prompt design influences this encoding. It combines prompt engineering, LLM-derived summaries of Chandra X-ray source observations, and sparse autoencoders to probe the link between language and data-derived physics, showing that structured prompts significantly improve the alignment of textual embeddings with physical measurements. Notably, improvements in clustering purity for hardness ratio, power-law gamma, and variability index demonstrate that language can reflect underlying physics when prompts are carefully crafted, while SAEs reveal interpretable textual features associated with specific physical regimes. This work provides a pathway for extracting and interpreting physics from textual descriptions, enabling cross-modal analyses and informing future development of data-informed NLP tools in astrophysics.

Abstract

Large Language Models have demonstrated the ability to generalize well at many levels across domains, modalities, and even shown in-context learning capabilities. This enables research questions regarding how they can be used to encode physical information that is usually only available from scientific measurements, and loosely encoded in textual descriptions. Using astrophysics as a test bed, we investigate if LLM embeddings can codify physical summary statistics that are obtained from scientific measurements through two main questions: 1) Does prompting play a role on how those quantities are codified by the LLM? and 2) What aspects of language are most important in encoding the physics represented by the measurement? We investigate this using sparse autoencoders that extract interpretable features from the text.

Figures (4)

• Figure 1: This figure gives an overview of how this dataset was created. Starting with a Chandra observation, we queried the Chandra Archives for papers about each specific source within the observation, or each bright area within the yellow image. Then we fed these papers to gpt-4o-mini and asked it to create a summary about the physical characteristics of the source. The code also generated embeddings of these summaries and several of the physical parameters of these objects. Finally, we used these embeddings and summaries to create visuals of the physical information for interpretation. The two stars seen in this diagram emphasize what this work focuses on: prompt engineering and feature extraction.
• Figure 2: An example of visually understanding where the LLM is encoding physical information in the embeddings using t-SNE plots. The left plot is colored by hardness ratio, which is a broader summary statistic, with around 4000 samples. The right plot is in relation to power law gamma value with about 1700 samples, and the above text is an excerpt from a specific source summary within the circled cluster. Power law gamma is a specific spectral property, and while the explicit power law gamma value was never mentioned in the text summary, the LLM was able to infer and learn about this physical property from the language in the summary.
• Figure 3: t-SNE plot of source summaries colored by power law gamma. Each of the three isolated clusters was analyzed using SAEs to understand the underlying themes within the summaries that allowed them to cluster. The text by each summary highlights the themes found across the features.
• Figure 4: A label found by SAEs with the top words that activate for it. A similar feature was found across many clusters, and the semantic details within each feature changed for each differing dataset used as context.