Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Locating and Editing Figure-Ground Organization in Vision Transformers

Stefan Arnold, René Gröbner

TL;DR

It is found that figure-ground organization is ambiguous through early and intermediate layers and resolves abruptly in later layers, and it is shown that BEiT reliably favors convex completion under this competition.

Abstract

Vision Transformers must resolve figure-ground organization by choosing between completions driven by local geometric evidence and those favored by global organizational priors, giving rise to a characteristic perceptual ambiguity. We aim to locate where the canonical Gestalt prior convexity is realized within the internal components of BEiT. Using a controlled perceptual conflict based on synthetic shapes of darts, we systematically mask regions that equally admit either a concave completion or a convex completion. We show that BEiT reliably favors convex completion under this competition. Projecting internal activations into the model's discrete visual codebook space via logit attribution reveals that this preference is governed by identifiable functional units within transformer substructures. Specifically, we find that figure-ground organization is ambiguous through early and intermediate layers and resolves abruptly in later layers. By decomposing the direct effect of attention heads, we identify head L0H9 acting as an early seed, introducing a weak bias toward convexity. Downscaling this single attention head shifts the distributional mass of the perceptual conflict across a continuous decision boundary, allowing concave evidence to guide completion.

Locating and Editing Figure-Ground Organization in Vision Transformers

TL;DR

It is found that figure-ground organization is ambiguous through early and intermediate layers and resolves abruptly in later layers, and it is shown that BEiT reliably favors convex completion under this competition.

Abstract

Vision Transformers must resolve figure-ground organization by choosing between completions driven by local geometric evidence and those favored by global organizational priors, giving rise to a characteristic perceptual ambiguity. We aim to locate where the canonical Gestalt prior convexity is realized within the internal components of BEiT. Using a controlled perceptual conflict based on synthetic shapes of darts, we systematically mask regions that equally admit either a concave completion or a convex completion. We show that BEiT reliably favors convex completion under this competition. Projecting internal activations into the model's discrete visual codebook space via logit attribution reveals that this preference is governed by identifiable functional units within transformer substructures. Specifically, we find that figure-ground organization is ambiguous through early and intermediate layers and resolves abruptly in later layers. By decomposing the direct effect of attention heads, we identify head L0H9 acting as an early seed, introducing a weak bias toward convexity. Downscaling this single attention head shifts the distributional mass of the perceptual conflict across a continuous decision boundary, allowing concave evidence to guide completion.
Paper Structure (13 sections, 1 equation, 3 figures)

This paper contains 13 sections, 1 equation, 3 figures.

Table of Contents

  1. Introduction
  2. Contribution.
  3. Related Work
  4. Gestalt Laws in Human Perceptual Organization
  5. Emergent Gestalt Organization in Neural Networks
  6. Perceptual Conflict for Mechanistic Analysis
  7. Model Dissection via Logit Attribution
  8. Logit Attribution and Bias Emergence
  9. Attention Lens and Directional Pressure
  10. Model Steering via Activation Scaling
  11. Discussion and Conclusion
  12. Implications.
  13. Limitations.

Figures (3)

  • Figure 1: A visual stimulus of perceptual conflict. We define the conflict region (gray overlay) of interest by masking the patch-wise set difference between the concave shape (dark region) and its convex hull (red outline), inducing competition between local evidence and global priors.
  • Figure 2: Figure--ground preference across layers and heads. (a) Layer-wise logit attribution of the masked patch representation, showing signed residual-stream contributions favoring convex (positive) or concave (negative) completion across depth; points show medians with variability. (b) Head-wise logit attribution for attention, encoding the direction and magnitude of each head’s contribution, with marginal summaries indicating aggregate effects.
  • Figure :