Redefining Affordance via Computational Rationality

TL;DR

The paper addresses the ambiguity in traditional affordance theories by proposing a Computational Rationality (CR)–based framework in which affordances are constructed within an internal environment from bounded sensory input. Affordance perception becomes a decision problem guided by two core quantities: confidence in action execution and predicted utility of the outcome, with internal affordances learned and refined through reinforcement learning. The theory unifies ecological psychology, HCI/design, and ML perspectives by detailing mechanisms (feature recognition and hypothetical motion trajectories) and showcasing thought experiments that illustrate learning and adaptation across physical, digital, and social domains. The approach offers practical implications for designing adaptive, user-centered systems that evolve with user capabilities and experience.

Abstract

Affordances, a foundational concept in human-computer interaction and design, have traditionally been explained by direct-perception theories, which assume that individuals perceive action possibilities directly from the environment. However, these theories fall short of explaining how affordances are perceived, learned, refined, or misperceived, and how users choose between multiple affordances in dynamic contexts. This paper introduces a novel affordance theory grounded in Computational Rationality, positing that humans construct internal representations of the world based on bounded sensory inputs. Within these internal models, affordances are inferred through two core mechanisms: feature recognition and hypothetical motion trajectories. Our theory redefines affordance perception as a decision-making process, driven by two components: confidence (the perceived likelihood of successfully executing an action) and predicted utility (the expected value of the outcome). By balancing these factors, individuals make informed decisions about which actions to take. Our theory frames affordances perception as dynamic, continuously learned, and refined through reinforcement and feedback. We validate the theory via thought experiments and demonstrate its applicability across diverse types of affordances (e.g., physical, digital, social). Beyond clarifying and generalizing the understanding of affordances across contexts, our theory serves as a foundation for improving design communication and guiding the development of more adaptive and intuitive systems that evolve with user capabilities.

Figures (1)

• Figure 1: (a) Traditional affordance theory (Gibson's view): The organism directly perceives affordances as action possibilities presented by the environment, without the need for internal processing or cognitive mediation. These directly perceived affordances guide subsequent actions. (b) Computational Rationality-based affordance: Our theory proposes that while "external affordances" exist in the external environment, organisms can not directly perceive it. Instead, an internal representation (the "internal environment") is constructed from constrained sensory inputs. Within this internal environment, "internal affordances" are inferred, which contain two components: confidence in executing the action and predicted utility (expected outcome of taking an action). The organism refines these internal affordances through reinforcement learning, using feedback from previous actions (e.g., rewards) to continuously update and improve affordance inferences.