Rethinking the Zigzag Flattening for Image Reading

TL;DR

The paper addresses how to read 2D images in CV models, challenging the default zigzag flattening and proposing Hilbert fractal flattening (HF) as a locality-preserving alternative. It provides a theoretical framework for HF, including dilation factors and a DeGrid metric, and introduces Arbitrary Hilbert Flattening for flexible grid sizes. Empirically, HF improves image scaling robustness and boosts performance for MLP-based architectures (notably MLP-Mixer and FPN-MLPs) on CIFAR-10, validating HF as a practical replacement for ZF. Overall, HF offers a principled, plug-in flattening strategy that better preserves spatial locality across scales, with meaningful improvements in multi-scale vision models.

Abstract

Sequence ordering of word vector matters a lot to text reading, which has been proven in natural language processing (NLP). However, the rule of different sequence ordering in computer vision (CV) was not well explored, e.g., why the ``zigzag" flattening (ZF) is commonly utilized as a default option to get the image patches ordering in vision networks. Notably, when decomposing multi-scale images, the ZF could not maintain the invariance of feature point positions. To this end, we investigate the Hilbert fractal flattening (HF) as another method for sequence ordering in CV and contrast it against ZF. The HF has proven to be superior to other curves in maintaining spatial locality, when performing multi-scale transformations of dimensional space. And it can be easily plugged into most deep neural networks (DNNs). Extensive experiments demonstrate that it can yield consistent and significant performance boosts for a variety of architectures. Finally, we hope that our studies spark further research about the flattening strategy of image reading.

Figures (8)

• Figure 1: Illustration of Image Reading.
• Figure 2: Multi-scale transformation of dimensional space with Zigzag curve and Hilbert curve flattening/folding, respectively. Specifically, suppose that two points separated by $d_L \in (0^+, 1)$ are taken at random on the real number axis in the interval $I = [0,1]$, which can always be taken if $2^n\times 2^n$ ($n \in \{3,4,5,6,7\}$) points are equally spaced on the number axis. By dividing the interval of the numerical axis by $2^6$ equal parts and using different unfolding operations (ZF and HF), we find that the distance $d_L$ between two points changes to $d_Z^6$ and $d_H^6$. Similarly, if $2^8$ equal divisions are performed, the distance between the two points again changes to $d_Z^8$ and $d_H^8$. The two points with fixed distance $d_L$ on $[0,1]$ mapped to 2-D space under different types of unfolding operations will have various distances.
• Figure 3: Zigzag flattening (red lines and arrows) VS. Hilbert flattening (blue lines and arrows) in ViT for image patch expanding. When flattening a 2D image into a 1D patch sequence, ZF will move the initially adjacent image blocks (semantically related patches) away from each other, but HF does not. That is, the head of cat remained clustered together after slicing at different flattening scales, and the position of the head on the 1D sequence was not change.
• Figure 4: Suppose we compute the DeGrid at each point on a $8 \times8$ grid structure data with $K=2$. The results are expressed as a heat map, in which the brightness of the pixels indicates the grid structure info preservation. The closer the pixels are to black, the better.
• Figure 5: Thanks to pseudoarbitrary, we present here examples of arbitrary Hilbert flattening. The left panel shows a resolution of $14\times14$, while the right panel at $16\times9$.

Theorems & Definitions (6)

• Definition 3.1
• Theorem 3.2
• Remark 3.3
• Remark 3.4
• Definition 3.5
• Corollary 3.6