Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Coordinate Prediction Bias from Positional Encoding Failures

Xingjian Tao, Yiwei Wang, Yujun Cai, Yihong Luo, Jing Tang

TL;DR

This work identifies a systematic, directional bias in coordinate predictions by multimodal LLMs when visual positional encodings degrade at high resolutions. It introduces Vision-PE Shuffle Guidance (VPSG), a training-free test-time method that contrasts a position-conditioned decoding path with multiple shuffled-PE auxiliary paths to generate negative evidence and suppress spurious digits via a final-layer logit adjustment. Through causal analysis and ablations, VPSG proves robust, relying on seeds-aggregation and a geometry-based decay schedule to concentrate corrections on the most influential digits while preserving formatting. Experiments on ScreenSpot-Pro show consistent improvements across model scales without retraining, underscoring the importance of robust positional encoding for precise spatial reasoning in vision–language systems.

Abstract

Multimodal large language models (MLLMs) excel at vision-language tasks such as VQA and document understanding, yet precise coordinate prediction remains challenging. High-resolution inputs exacerbate this difficulty by producing long token sequences that weaken positional encodings and introduce directional biases in coordinate outputs. We investigate this phenomenon by analyzing how MLLMs behave when visual positional encodings (VPEs) are deliberately perturbed through shuffling. Our analysis reveals that such perturbations induce predictable, non-random coordinate biases rather than random errors, suggesting that models rely on internal positional priors when spatial grounding signals are degraded. Crucially, we observe similar directional error patterns in natural high-resolution datasets, indicating that positional encoding failures are a key bottleneck for accurate coordinate prediction at scale. To address this issue, we propose Vision-PE Shuffle Guidance (VPSG), a training-free test-time method that leverages the directional nature of these biases for correction. VPSG runs auxiliary decoding with shuffled VPEs to isolate position-unconditioned tendencies, then uses this as negative evidence to guide digit prediction while preserving coordinate format through a lightweight finite-state machine. Experiments on ScreenSpot-Pro demonstrate reliable improvements, highlighting positional encoding robustness as a critical factor for spatial reasoning in MLLMs.

Mitigating Coordinate Prediction Bias from Positional Encoding Failures

TL;DR

This work identifies a systematic, directional bias in coordinate predictions by multimodal LLMs when visual positional encodings degrade at high resolutions. It introduces Vision-PE Shuffle Guidance (VPSG), a training-free test-time method that contrasts a position-conditioned decoding path with multiple shuffled-PE auxiliary paths to generate negative evidence and suppress spurious digits via a final-layer logit adjustment. Through causal analysis and ablations, VPSG proves robust, relying on seeds-aggregation and a geometry-based decay schedule to concentrate corrections on the most influential digits while preserving formatting. Experiments on ScreenSpot-Pro show consistent improvements across model scales without retraining, underscoring the importance of robust positional encoding for precise spatial reasoning in vision–language systems.

Abstract

Multimodal large language models (MLLMs) excel at vision-language tasks such as VQA and document understanding, yet precise coordinate prediction remains challenging. High-resolution inputs exacerbate this difficulty by producing long token sequences that weaken positional encodings and introduce directional biases in coordinate outputs. We investigate this phenomenon by analyzing how MLLMs behave when visual positional encodings (VPEs) are deliberately perturbed through shuffling. Our analysis reveals that such perturbations induce predictable, non-random coordinate biases rather than random errors, suggesting that models rely on internal positional priors when spatial grounding signals are degraded. Crucially, we observe similar directional error patterns in natural high-resolution datasets, indicating that positional encoding failures are a key bottleneck for accurate coordinate prediction at scale. To address this issue, we propose Vision-PE Shuffle Guidance (VPSG), a training-free test-time method that leverages the directional nature of these biases for correction. VPSG runs auxiliary decoding with shuffled VPEs to isolate position-unconditioned tendencies, then uses this as negative evidence to guide digit prediction while preserving coordinate format through a lightweight finite-state machine. Experiments on ScreenSpot-Pro demonstrate reliable improvements, highlighting positional encoding robustness as a critical factor for spatial reasoning in MLLMs.
Paper Structure (36 sections, 2 theorems, 25 equations, 6 figures, 7 tables, 1 algorithm)

This paper contains 36 sections, 2 theorems, 25 equations, 6 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Coordinate prediction with MLLMs
  4. Visual position encoding
  5. Methodology
  6. Causal view of coordinate prediction
  7. Bias analysis
  8. Overall algorithm.
  9. Seeds aggregation.
  10. Coefficient decay.
  11. Summary.
  12. Experiments
  13. Experimental settings
  14. Datasets.
  15. Models.
  16. ...and 21 more sections

Key Result

Proposition 1

Let $\mathcal{D}\subset\mathcal{V}$ denote the digit subset of the vocabulary and $\alpha_t$ the step-wise guidance coefficient determined by the FSM. The VPSG-adjusted distribution satisfies where $\tilde{\ell}_B(v)=\log p_B(v)$ is the log-probability of the position-unconditioned reference.

Figures (6)

  • Figure 1: Effect of shuffling visual positional encodings: removing spatial conditioning causes the model to collapse to similar coordinate predictions across independent runs, indicating a position-unconditioned, directional bias rather than random variation. We also observe a similar clustered error pattern on high-resolution images (without shuffling), consistent with position-encoding failures.
  • Figure 2: Causal view of coordinate prediction. Image content and prompt provide the intended causal effect on output digits (left). When VPEs are missing (e.g., at high resolution), the model relies on spurious correlations, leading to directional digit biases (right).
  • Figure 3: Diagonal-normalized average pairwise distance $\tilde{d}$ between coordinate predictions under Normal PEs and Shuffled PEs.
  • Figure 4: We obtain a position-unconditioned reference by shuffling visual positional encodings (left) and fuse it with the position-conditioned prediction (right). This conditional–unconditional contrast serves as negative evidence for digits, strengthening positional cues and suppressing spurious numeric patterns during decoding.
  • Figure 5: Qualitative example of VPSG on a Screenspot-Pro case. The base model with normal positional encodings produces a biased coordinate prediction ([1024, 512]), while the same model with shuffled positional encodings collapses to a similar but consistently biased point ([1024, 356]), revealing a directional position-unconditioned tendency. Applying VPSG corrects this bias and outputs the accurate ground truth ([659, 857]) by integrating negative evidence from multiple shuffled runs and reweighting digit logits, demonstrating how VPSG suppresses spurious patterns and restores faithful spatial grounding.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Proposition 1: VPSG token guidance
  • Lemma 1: Argmax invariance under positive affine transforms
  • proof