TextBandit: Evaluating Probabilistic Reasoning in LLMs Through Language-Only Decision Tasks

TL;DR

TextBandit presents a text-only multi-armed bandit benchmark to probe whether LLMs can perform probabilistic reasoning under uncertainty using solely linguistic feedback. By comparing four open-source LLMs against classical decision-making baselines, the study reveals substantial heterogeneity in language-driven learning, with Qwen3-4B attaining a dominant best-arm selection rate of 89.2%, suggesting that probabilistic reasoning can emerge from language alone. The work highlights that model size alone does not guarantee better decision performance and that architectural and inference dynamics shape how language models exploit textual cues for exploration and exploitation. This benchmark provides a naturalistic, non-numeric context for evaluating decision-making capabilities in LLMs and points to future directions involving dynamic environments, delayed feedback, and targeted prompting to harness language-based probabilistic reasoning in practical applications.

Abstract

Large language models (LLMs) have shown to be increasingly capable of performing reasoning tasks, but their ability to make sequential decisions under uncertainty only using natural language remains underexplored. We introduce a novel benchmark in which LLMs interact with multi-armed bandit environments using purely textual feedback, "you earned a token", without access to numerical cues or explicit probabilities, resulting in the model to infer latent reward structures purely off linguistic cues and to adapt accordingly. We evaluated the performance of four open-source LLMs and compare their performance to standard decision-making algorithms such as Thompson Sampling, Epsilon Greedy, Upper Confidence Bound (UCB), and random choice. While most of the LLMs underperformed compared to the baselines, Qwen3-4B, achieved the best-arm selection rate of 89.2% , which significantly outperformed both the larger LLMs and traditional methods. Our findings suggest that probabilistic reasoning is able to emerge from language alone, and we present this benchmark as a step towards evaluating decision-making capabilities in naturalistic, non-numeric contexts.

Figures (2)

• Figure 1: Comparison of cumulative regret trends for four LLMs.
• Figure 2: Comparison of reward trends across four LLMs in the bandit task.