Hot-Start from Pixels: Low-Resolution Visual Tokens for Chinese Language Modeling

TL;DR

This work questions the primacy of index-based tokenization for Chinese language modeling by leveraging the visual structure of logographic characters. It introduces a vision_token pipeline that renders characters as low_resolution grayscale images processed by a CNN-based visual encoder before feeding a GPT_2_small_style decoder, and contrasts it with a traditional token ID path. Empirically, 8x8 grayscale inputs achieve 39.2% next-character accuracy, on par with the 39.1% index baseline, and a pronounced hot_start effect accelerates learning in the visual pathway. The approach remains robust across resolutions and partial visibility, with interpretable embedding geometry and gradient analyses linking predictions to salient visual regions. Overall, the results suggest visual glyphs provide a sample-efficient inductive bias for Chinese LM, offering a complementary and potentially more interpretable representation to token-based methods.

Abstract

Large language models typically represent Chinese characters as discrete index-based tokens, largely ignoring their visual form. For logographic scripts, visual structure carries semantic and phonetic information, which may aid prediction. We investigate whether low-resolution visual inputs can serve as an alternative for character-level modeling. Instead of token IDs, our decoder receives grayscale images of individual characters, with resolutions as low as $8 \times 8$ pixels. Remarkably, these inputs achieve 39.2\% accuracy, comparable to the index-based baseline of 39.1\%. Such low-resource settings also exhibit a pronounced \emph{hot-start} effect: by 0.4\% of total training, accuracy reaches above 12\%, while index-based models lag at below 6\%. Overall, our results demonstrate that minimal visual structure can provide a robust and efficient signal for Chinese language modeling, offering an alternative perspective on character representation that complements traditional index-based approaches.

Figures (5)

• Figure 1: Model architecture and concrete processing example showing the prediction of the final character in the phrase "数据显示" (The data shown); numerical results demonstrate the hot-start phenomenon of visual tokens by 0.5% of the total training progress. Top: Visual-based training pipeline; numerical results are based on inputs of $8\times8$ character images. Bottom: Index-based training pipeline.
• Figure 2: Heatmap visualization of character example cropping at two resolutions: low and high.
• Figure 3: "toast-center effect": center strokes (blue box) receive more attention than outer pixels (red box).
• Figure 4: Validation accuracy across image resolutions at selected early training checkpoints (5k--10k samples), plotted on the index-based baseline scale (dashed line).
• Figure 5: PCA of vision-based embeddings reveals localized spatial groupings of visually similar Chinese characters.