What if Othello-Playing Language Models Could See?

TL;DR

This work questions whether grounding language models with vision aids learning in a symbolically structured domain. It introduces Vis-Othello, a multimodal model that jointly processes move sequences and board images via a VisualBERT-inspired training regime, including random and future token masking plus text-image prediction. The results show that multimodal learning improves sample efficiency and robustness to perturbations like board rotations, and that representations across modalities become increasingly aligned with more data. Overall, the study provides a controlled framework for analyzing grounded representations and cross-modal transfer, with implications for multimodal reasoning in other structured tasks.

Abstract

Language models are often said to face a symbol grounding problem. While some have argued the problem can be solved without resort to other modalities, many have speculated that grounded learning is more efficient. We explore this question in Othello, a simplified, rule-based world that offers a controlled and interpretable testbed for studying world understanding. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained jointly on move sequences and board images. Using the Othello rule understanding task, we examine whether multi-modal learning provides advantages over text-only approaches. We further evaluate robustness under semantically irrelevant perturbations and analyze the consistency of cross-modal alignment. Our results suggest that multi-modal training not only improves performance and robustness but also promotes convergence toward shared internal representations across different model architectures.

Figures (4)

• Figure 1: Architecture of Vis-Othello. The model integrates visual and textual inputs by encoding board images and corresponding move sequences using a Transformer. During pretraining, (i) a ResNet is trained to predict the next move from the current board image; (ii) the multi-modal Transformer is pretrained with three objectives: text-image prediction, random token masking, and future token masking.
• Figure 2: Illustration of probing results for BERT and Vis-Othello trained with different dataset set sizes.
• Figure 3: Illustration of Rotation $180^{\circ}$. A $180^{\circ}$ rotation preserves game dynamics due to the board’s inherent symmetry and the uniformity of move rules, making such transformations invariant under play.
• Figure 4: Comparison of models' performance with and without board rotation across different training dataset sizes. The results demonstrate that multi-modal models maintain better performance under rotation compared to purely visual models. $-P$ indicates the model is pretrained, while $-S$ indicates it is trained from scratch.