Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding

TL;DR

This paper revisits the fundamental building block of visual foundation models—the tokenizer—arguing that patch-based patches poorly align with object semantics in remote sensing. It proposes HOOK, a homogeneous visual tokenizer with two modules: Object Perception Module (OPM) and Object Vectorization Module (OVM). HOOK constructs semantically independent regions (SIRs) from $4\times4$ seeds and outputs an adaptable number of tokens via $N$ learnable queries, achieving the ideal of 'same object same token'. Theoretical analysis via a token-object confusion matrix and two routing strategies informs HOOK's design, which demonstrates strong performance and efficiency on sparse classification and dense segmentation across multiple remote-sensing datasets, while remaining image- and task-agnostic. Limitations include local attention bounds and color/texture confounds, with future work aiming to refine SIR splitting and explore alternative routing routes to further improve homogeneity and adaptability.

Abstract

The tokenizer, as one of the fundamental components of large models, has long been overlooked or even misunderstood in visual tasks. One key factor of the great comprehension power of the large language model is that natural language tokenizers utilize meaningful words or subwords as the basic elements of language. In contrast, mainstream visual tokenizers, represented by patch-based methods such as Patch Embed, rely on meaningless rectangular patches as basic elements of vision, which cannot serve as effectively as words or subwords in language. Starting from the essence of the tokenizer, we defined semantically independent regions (SIRs) for vision. We designed a simple HOmogeneous visual tOKenizer: HOOK. HOOK mainly consists of two modules: the Object Perception Module (OPM) and the Object Vectorization Module (OVM). To achieve homogeneity, the OPM splits the image into 4*4 pixel seeds and then utilizes the attention mechanism to perceive SIRs. The OVM employs cross-attention to merge seeds within the same SIR. To achieve adaptability, the OVM defines a variable number of learnable vectors as cross-attention queries, allowing for the adjustment of token quantity. We conducted experiments on the NWPU-RESISC45, WHU-RS19 classification dataset, and GID5 segmentation dataset for sparse and dense tasks. The results demonstrate that the visual tokens obtained by HOOK correspond to individual objects, which demonstrates homogeneity. HOOK outperformed Patch Embed by 6\% and 10\% in the two tasks and achieved state-of-the-art performance compared to the baselines used for comparison. Compared to Patch Embed, which requires more than one hundred tokens for one image, HOOK requires only 6 and 8 tokens for sparse and dense tasks, respectively, resulting in efficiency improvements of 1.5 to 2.8 times. The code is available at https://github.com/GeoX-Lab/Hook.

Figures (9)

• Figure 1: (a) A natural language tokenizer uses words or subwords as the basic elements of language. Similarly, for visual tokenizers, this work attempts to answer a fundamental question: What are the basic elements of vision? (b) The current mainstream visual tokenizers are patch-based methods, represented by Patch Embed, which can be categorized into three types on the basis of hierarchy: patch level, subpatch level, and superpatch level. Their commonality lies in the use of patches as the basic elements of vision. (c) There exists a confusion matrix for tokens and objects. The "same object multiple tokens" leads to incomplete object features, whereas the "same token multiple objects" leads to unclear relationships between objects. The "multiple tokens multiple objects" inherits the drawbacks of the above two approaches. Patch-based methods inherently struggle to achieve the ideal "same token same object" scenario. (d) We define the concept of semantically independent region (SIRs) and propose two properties of an ideal visual tokenizer: homogeneity and adaptability. We design a simple HOmogeneous visual tOKenizer, HOOK, where SIRs as the basic elements of vision.
• Figure 2: Confusion matrix of tokens and objects. In a simplified scenario and under strict definitions, the confusion matrix of tokens and objects reveals two general routes for constructing homogeneous visual tokens: (1) splitting and merging and (2) merging and splitting.
• Figure 3: The architecture of HOOK consists of two modules: the object perception module (OPM), which is responsible for perceiving semantically independent regions, and the object vectorization module (OVM), which is responsible for vectorizing semantically independent regions into tokens.
• Figure 4: HOOK is capable of adapting to tasks with different granularities. For sparse tasks represented by classification, the model can average the tokens and then pass them through a linear classification layer to output the classification results. For dense tasks represented by segmentation, the model can utilize the intermediate variable, namely, the attention map from the OVM, to restore the number of tokens and then pass them to the segmentation head to output the segmentation results.
• Figure 5: Visualization results for homogeneous visual tokens. The left image is the original image, whereas the right image displays the regions of the token, with each colour representing one region of the token. The images in (a) are obtained from the NWPU-RESISC45 classification dataset, whereas the images in (b) are obtained from the GID5 semantic segmentation dataset.

Theorems & Definitions (8)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6
• Definition 7
• Definition 8