When Text-as-Vision Meets Semantic IDs in Generative Recommendation: An Empirical Study

TL;DR

This work addresses how Semantic IDs for Generative Recommendation can benefit from OCR-based visual text representations instead of standard text encodings, especially when item descriptions are symbol-rich and attribute-centric. By rendering descriptions as images and using OCR encoders, the study conducts extensive experiments across four datasets and two backbones in both unimodal and multimodal settings, across multiple fusion strategies. Results show OCR-text often matches or surpasses traditional text embeddings, with the largest improvements on datasets featuring dense symbolic attributes, and demonstrates strong alignment with item image embeddings and robustness to rendering quality and OCR choices. The findings suggest vision-centric text representations as a practical, robust alternative for semantic ID learning in real-world, multimodal recommendation systems.

Abstract

Semantic ID learning is a key interface in Generative Recommendation (GR) models, mapping items to discrete identifiers grounded in side information, most commonly via a pretrained text encoder. However, these text encoders are primarily optimized for well-formed natural language. In real-world recommendation data, item descriptions are often symbolic and attribute-centric, containing numerals, units, and abbreviations. These text encoders can break these signals into fragmented tokens, weakening semantic coherence and distorting relationships among attributes. Worse still, when moving to multimodal GR, relying on standard text encoders introduces an additional obstacle: text and image embeddings often exhibit mismatched geometric structures, making cross-modal fusion less effective and less stable. In this paper, we revisit representation design for Semantic ID learning by treating text as a visual signal. We conduct a systematic empirical study of OCR-based text representations, obtained by rendering item descriptions into images and encoding them with vision-based OCR models. Experiments across four datasets and two generative backbones show that OCR-text consistently matches or surpasses standard text embeddings for Semantic ID learning in both unimodal and multimodal settings. Furthermore, we find that OCR-based Semantic IDs remain robust under extreme spatial-resolution compression, indicating strong robustness and efficiency in practical deployments.

Figures (6)

• Figure 1: Embedding geometry across modalities. We project three item representations into a shared 2D space: Item image emb, extracted from each item’s photos; OCR-based text emb, extracted by rendering the item’s textual description into an image and encoding it with an OCR model; and Standard text emb, produced by a standard text encoder operating on item’s textual description. Each point denotes an item, and cross-modal lines connect representations of the same item. OCR-text embeddings form smooth manifolds that align closely with image embeddings, whereas text embeddings exhibit higher dispersion and stronger anisotropy.
• Figure 2: Conceptual illustration of representation spaces induced by different encoders.
• Figure 3: An overview of the Semantic ID-based GR pipeline.
• Figure 4: Recommendation Performance of Multimodal Semantic IDs Constructed with the Early-Fusion.
• Figure 5: Recommendation Performance of Multimodal Semantic IDs Constructed with Three Late-Fusion Strategies.