Smart Walkers in Discrete Space

TL;DR

The paper studies a two-walker chaser-target system on a 1D discrete space, deriving analytic descriptions for the first-encounter distribution $\bm{\mathcal{P}}$ and mean meeting time $\tau_{a,b}$ in the baseline random-walker case via an absorbing Markov framework. It then introduces a Smart Walker trained with Q-learning on the joint state space, producing a non-factorizable global transition matrix that reshapes encounter statistics and enables computation of the same observables, along with thermodynamic and policy entropies $S_T$ and $S_S$. The authors show that different reward structures induce distinct learned policies, with time-dependent rewards yielding the strongest information encoding and sinusoidal rewards closest to random; $S_T$ correlates with $S_S$ and serves as a post hoc proxy for learned skill. They further validate this proxy by evaluating configuration entropy against Stockfish skill levels, observing a clear relationship and a notable discontinuity at the highest level, suggesting $S_T$ captures qualitative shifts in agent ability. The findings bridge stochastic pursuit dynamics and reinforcement learning, proposing configuration entropy as a broadly applicable tool for assessing intelligent behavior when explicit policies are inaccessible.

Abstract

We study the statistical properties of trainable agents moving in discrete space. After introducing the mathematical framework, we first analyze the dynamics of two completely random walkers, mutually competing in a chaser-target interaction scheme. The statistics of the encounters is analytically obtained and the predictions tested versus numerical simulations. We then move forward to extend the baseline case to agents capable of learning and adapting to an external reward signal, using reinforcement learning. Smart walkers morph the statistics of the encounter, to maximize their cumulated reward, as confirmed by combined numerical and analytical insights. More interestingly, configuration entropy proves a reliable proxy to gauge the acquired ability of the agents to cope with the assigned task when no other information about them (i.e. reward signal, policy, etc) is present. We further test the proposed measure of learned skills by operating the Stockfish chess engine against a quasi-random untrained opponent. The obtained conclusions corroborate our claim.

Figures (9)

• Figure 1: Illustration of the game environment for $N = 11$. Alice (orange) starts on the left, and Bob (blue) on the right. The scores displayed above and below the cells are drawn with the same color of the agent they refer to. Reflecting boundary conditions are applied at the edges. The depicted score distribution represents just one viable choice, among several other that we will set to explore all along the work.
• Figure 2: Transition matrices for the two completely random agents. Reflecting boundary conditions are encoded within the matrices. At this stage, there is no interaction between the walkers, which hence execute two independent random walks.
• Figure 3: First encounter probability distribution for two random walkers under reflecting boundary conditions, with $N = 11$ and cell indices starting from $0$ ($50{,}000$ simulated episodes). In the limit of infinitely many simulations, the empirical and theoretical distributions converge. Note that the tails of the distribution decrease approximately linearly, indicating a behavior that deviates from a quadratic profile.
• Figure 4: Visualization of the vector $\bm{t}$ representing meeting times for two random walkers on a discrete space with $N = 11$. The left plot depicts directly the values of the elements of $\bm{t}$. The right plot instead shows a factorized view of $\bm{t}$ across the individual walker subspaces using Eq. \ref{['eq:closed-form-t']}, resulting in a bar plot of mean meeting times (in game moves) as a function of the walkers' starting positions. The central trough indicates zero encounter times when both walkers start from the same cell.
• Figure 5: Generalized game environment for smart walkers. Each agent is equipped with a Q-table, which can be mapped to a corresponding policy tensor via Eq. \ref{['eq:temperature_softmax']}. Initially, both policies correspond to a random walk. Over repeated games, Alice updates her Q-table using Eq. \ref{['eq:q-learning']}, thereby modifying her policy tensor and inducing a new dynamic. Both agents’ policies can be used to construct the global transition matrix $A$, as detailed in Appendix \ref{['app:get_global_A_from_policy']}.