"Just in Time" World Modeling Supports Human Planning and Reasoning

TL;DR

The paper tackles how humans reason and plan in complex environments despite cognitive limits. It introduces the Just-in-Time (JIT) model, which interleaves simulation with online encoding via a visual lookahead to build task-relevant construals $C \,\subseteq\, O$ while obeying a memory-forgetting rule $p(\text{forget } o) \propto t^{−\gamma}$. Compared to Value Guided Construal (VGC), JIT emphasizes need-based encoding rather than globally optimizing construal utility, and it is tested in grid-world planning and Plinko-like physical prediction tasks. Across planning and physics experiments, JIT better accounts for human memory and attention patterns and demonstrates efficient representations that support accurate predictions with limited encoding; the work suggests a general, scalable mechanism for cognitive construal formation with implications for AI and robotics. The findings highlight the potential of incremental construal construction as a robust strategy for managing cognitive load in real-world reasoning tasks, while noting limitations and directions for integrating prior knowledge and richer scene complexity.

Abstract

Probabilistic mental simulation is thought to play a key role in human reasoning, planning, and prediction, yet the demands of simulation in complex environments exceed realistic human capacity limits. A theory with growing evidence is that people simulate using simplified representations of the environment that abstract away from irrelevant details, but it is unclear how people determine these simplifications efficiently. Here, we present a "Just-in-Time" framework for simulation-based reasoning that demonstrates how such representations can be constructed online with minimal added computation. The model uses a tight interleaving of simulation, visual search, and representation modification, with the current simulation guiding where to look and visual search flagging objects that should be encoded for subsequent simulation. Despite only ever encoding a small subset of objects, the model makes high-utility predictions. We find strong empirical support for this account over alternative models in a grid-world planning task and a physical reasoning task across a range of behavioral measures. Together, these results offer a concrete algorithmic account of how people construct reduced representations to support efficient mental simulation.

Figures (28)

• Figure 1: The Just-in-Time model for constructing representations for planning and reasoning, and the two test domains. Left: a grid world task where the goal is to navigate an agent from start (blue circle) to goal (green square). Right: a physical prediction task, where the goal is to predict where a ball will land after falling through an array of obstacles. JIT incrementally updates a representation in between steps of simulation and perceptual lookahead. At each step of a model run, the state of the simulation is first incremented (simulate). Then, this state is used to guide a visual search of the scene to find potential collisions (lookahead), and any objects flagged by this search are then used included in the representation (encode), to drive the next steps of simulation. In the planning domain, simulation is implemented as a stochastic variant of the $A^\star$ search algorithm zhi2020online, and in the physics domain, simulation is implemented as a probabilistic physics simulation engine battaglia_simulation_2013
• Figure 2: Comparisons between human results, Just-in-Time model, and the Value-Guided Construal model in the process-tracing experiments (1D and 1E) of ho2022people. Participants were shown a blank maze in which only the center object was initially visible. They were instructed that moving the mouse over a masked object would reveal it. Their task was to plan a path from the start (blue circle) to the goal (highlighted green square), revealing as many objects as needed along the way. Heatmaps show the probability of an object being revealed by people, and the representational weight JIT and VGC assign to each object.
• Figure 3: (A) Predictions from our JIT model (top row) compared to value-guided construal (bottom row) against human responses for three experiments in ho2022people. The blue line denotes the identity line, and the gray line denotes the line of best fit. (B) Quantitative model comparisons of compute and memory resource use for a set of procedurally generated grid worlds. We calculate a resource efficiency measure for four models (JIT compared to planning with the maximal representation, planning with a random representation, and planning with VGC) under a variety of cost assumptions. The net utility is inversely proportional to the sum of costs of representation (y-axis), computation (x-axis), and the length of the resulting plan. Heatmap colors visualize the most efficient model for each parameter regime. (C) Average proportion of objects represented by JIT and VGC in the same procedurally generated worlds.
• Figure 4: (A) Design of Experiment 2A. Participants saw a red ball above an array of obstacles and made predictions about where it would land. On a subset of trials, participants were then subsequently probed for their memory of a randomly chosen object's position. (B) Scatterplot of model predictions against human memory for objects. The blue dashed line denotes the identity line, and gray dashed lines denote the line of best fit. Error bars denote standard error. (C) Scatterplot of model predictions against confidence in recall (both z-scored).
• Figure 5: Selected worlds from Experiment 2A and 2B, where participants predicted the path that a red ball will take when let go, and were subsequently probed for memory of selected objects in the scene. We fit model parameters to maximize correlation with recall from experiment 2A, and transfer those parameters as-is to experiment 2B. Heatmaps show human recall for probed objects and predicted representations from both JIT and VGC; we did not probe memory for the gray hatched objects.