Optimizing the Landscape of LLM Embeddings with Dynamic Exploratory Graph Analysis for Generative Psychometrics: A Monte Carlo Study

TL;DR

The paper reframes LLM embeddings as navigable landscapes and demonstrates that embedding depth critically shapes psychometric structure. By coupling Dynamic Exploratory Graph Analysis (DynEGA) with a composite optimization of Normalized Mutual Information and Total Entropy Fit Index, it identifies depth regions that jointly maximize structural recovery and within-dimension coherence across item pools. Large-scale simulations on five grandiose narcissism dimensions show that single-metric optimization yields suboptimal, inconsistent solutions, while landscape-guided depth selection consistently improves dimensional accuracy and interpretability. The work highlights embedding non-uniformity, advocates composite, model-aware optimization, and positions embedding structure as substantive validity evidence for generative psychometrics with practical implications for item generation and construct definition.

Abstract

Large language model (LLM) embeddings are increasingly used to estimate dimensional structure in psychological item pools prior to data collection, yet current applications treat embeddings as static, cross-sectional representations. This approach implicitly assumes uniform contribution across all embedding coordinates and overlooks the possibility that optimal structural information may be concentrated in specific regions of the embedding space. This study reframes embeddings as searchable landscapes and adapts Dynamic Exploratory Graph Analysis (DynEGA) to systematically traverse embedding coordinates, treating the dimension index as a pseudo-temporal ordering analogous to intensive longitudinal trajectories. A large-scale Monte Carlo simulation embedded items representing five dimensions of grandiose narcissism using OpenAI's text-embedding-3-small model, generating network estimations across systematically varied item pool sizes (3-40 items per dimension) and embedding depths (3-1,298 dimensions). Results reveal that Total Entropy Fit Index (TEFI) and Normalized Mutual Information (NMI) leads to competing optimization trajectories across the embedding landscape. TEFI achieves minima at deep embedding ranges (900--1,200 dimensions) where entropy-based organization is maximal but structural accuracy degrades, whereas NMI peaks at shallow depths where dimensional recovery is strongest but entropy-based fit remains suboptimal. Single-metric optimization produces structurally incoherent solutions, whereas a weighted composite criterion identifies embedding dimensions depth regions that jointly balance accuracy and organization. Optimal embedding depth scales systematically with item pool size. These findings establish embedding landscapes as non-uniform semantic spaces requiring principled optimization rather than default full-vector usage.

Figures (6)

• Figure 1: Example of the embedding landscape search illustrating NMI (red line) and TEFI (blue line) trajectories across embedding dimensions and the resulting composite optimum (dashed vertical green line).
• Figure 2: Vector field of TEFI-NMI dynamics across the embedding landscape, with arrows indicating local GLLA-based first-order derivatives, color denoting item count (3 items = dark blue, 40 items = yellow), and thickness reflecting embedding position.
• Figure 3: Normalized Mutual Information (NMI; y-axis) across embedding depth by number of items per dimension (color), and number of embedding dimensions used (x-axis). NMI measures correspondence between estimated and generating dimensional structures (0 = no correspondence, 1 = perfect recovery). Color gradient indicates item pool size (dark blue = 3 items per dimension, yellow = 40 items per dimension)
• Figure 4: Total Entropy Fit Index (TEFI) across embedding depth by item pool size. Lower (more negative) TEFI values indicate better entropy-based fit and greater structural organization. Color gradient indicates item pool size (dark blue = 3 items per dimension, yellow = 40 items per dimension).
• Figure 5: Comparison of mean embedding dimension depth (y-axis) per number of items (x-axis) optimized via total entropy fit index (red circles) and normalized mutual information (blue circles).