Context-Dependent Affordance Computation in Vision-Language Models

TL;DR

These findings establish that VLMs compute affordances in a substantially context-dependent manner -- with the difference between lexical (90%) and semantic (58.5%) measures reflecting that surface vocabulary changes more than underlying meaning under context shifts -- and suggest a direction for robotics research: dynamic, query-dependent ontological projection (JIT Ontology) rather than static world modeling.

Abstract

We characterize the phenomenon of context-dependent affordance computation in vision-language models (VLMs). Through a large-scale computational study (n=3,213 scene-context pairs from COCO-2017) using Qwen-VL 30B and LLaVA-1.5-13B subject to systematic context priming across 7 agentic personas, we demonstrate massive affordance drift: mean Jaccard similarity between context conditions is 0.095 (95% CI: [0.093, 0.096], p < 0.0001), indicating that >90% of lexical scene description is context-dependent. Sentence-level cosine similarity confirms substantial drift at the semantic level (mean = 0.415, 58.5% context-dependent). Stochastic baseline experiments (2,384 inference runs across 4 temperatures and 5 seeds) confirm this drift reflects genuine context effects rather than generation noise: within-prime variance is substantially lower than cross-prime variance across all conditions. Tucker decomposition with bootstrap stability analysis (n=1,000 resamples) reveals stable orthogonal latent factors: a "Culinary Manifold" isolated to chef contexts and an "Access Axis" spanning child-mobility contrasts. These findings establish that VLMs compute affordances in a substantially context-dependent manner -- with the difference between lexical (90%) and semantic (58.5%) measures reflecting that surface vocabulary changes more than underlying meaning under context shifts -- and suggest a direction for robotics research: dynamic, query-dependent ontological projection (JIT Ontology) rather than static world modeling. We do not claim to establish processing order or architectural primacy; such claims require internal representational analysis beyond output behavior.

Figures (3)

• Figure 1: Comparison of visual processing pipelines. (a) Standard computer vision computes geometry before semantics, producing a fixed scene ontology. (b) The proposed Semantic-First architecture conditions geometric processing on agent context $\Theta$, enabling dynamic, task-relevant representations. Arrow direction indicates computational dependency.
• Figure 2: Distribution of pairwise Jaccard similarity between context primes ($n=9{,}244$ pairs). Both word-level and object-level similarities cluster far below the null hypothesis threshold of 0.5, with observed means of 0.095 and 0.119 respectively. The gap $\Delta = 0.405$ ($t = -674.72$, $p < 0.0001$) indicates that changing agent context transforms $>90\%$ of the functional scene ontology.
• Figure 3: Tucker decomposition factor loadings for context primes. Dim$_2$ reveals an isolated Culinary manifold where Chef (P1) loads at 0.95 while all other primes are near-zero or negative. Dim$_3$ captures an Access axis: Child (P3, +0.72) represents spatial openness/play, while Mobility (P4, $-$0.60) represents spatial constraint/obstruction. Dim$_1$ represents context-invariant salience, accounting for only 0.9% of variance.

Theorems & Definitions (4)

• Definition 2.1: Visual Field
• Definition 2.2: Agent State
• Definition 2.3: Affordance Mapping
• Definition 2.4: Action-Distance