TokenAR: Multiple Subject Generation via Autoregressive Token-level enhancement

TL;DR

This paper tackles identity confusion in autoregressive multi-reference image generation by introducing TokenAR, a token-level enhancement framework with Token Index Embedding, Identity-token Disentanglement, and Instruct Token Injection. It couples these techniques with the InstructAR Dataset, a large, open, multi-reference benchmark built via DreamRelation, automated masking, and rigorous VLM/DINOv2 filtering to ensure high identity fidelity and pose variation. Empirical results show TokenAR achieves state-of-the-art performance on both single- and multi-subject tasks, with ablations confirming the crucial roles of each component and a practical balance between model efficiency and generation quality. The work provides a practical path toward controllable, high-fidelity multi-subject image synthesis and releases substantial data and code for reproducibility.

Abstract

Autoregressive Model (AR) has shown remarkable success in conditional image generation. However, these approaches for multiple reference generation struggle with decoupling different reference identities. In this work, we propose the TokenAR framework, specifically focused on a simple but effective token-level enhancement mechanism to address reference identity confusion problem. Such token-level enhancement consists of three parts, 1). Token Index Embedding clusters the tokens index for better representing the same reference images; 2). Instruct Token Injection plays as a role of extra visual feature container to inject detailed and complementary priors for reference tokens; 3). The identity-token disentanglement strategy (ITD) explicitly guides the token representations toward independently representing the features of each identity.This token-enhancement framework significantly augments the capabilities of existing AR based methods in conditional image generation, enabling good identity consistency while preserving high quality background reconstruction. Driven by the goal of high-quality and high-diversity in multi-subject generation, we introduce the InstructAR Dataset, the first open-source, large-scale, multi-reference input, open domain image generation dataset that includes 28K training pairs, each example has two reference subjects, a relative prompt and a background with mask annotation, curated for multiple reference image generation training and evaluating. Comprehensive experiments validate that our approach surpasses current state-of-the-art models in multiple reference image generation task. The implementation code and datasets will be made publicly. Codes are available, see https://github.com/lyrig/TokenAR

Figures (13)

• Figure 1: Overview of InstructAR dataset, (a) Visualization of reference concept categories. Covering diverse reference concepts to insert including: toys, dogs, cats, persons, and plushies, (b) Visualization of relations in part of our datasets. There are 11 kinds of relations in total, with playing, hugging, and shaking the most, (c) Dataset construction process which contains three stages.
• Figure 2: Overview of TokenAR. Given different number of reference image, our approach process concatenation of multiple reference and a masked background through a frozen VAE Encoder to get visual tokens keeping their details. Text input, prompt, is encoded via text encoders to extract text guidance information. All the tokens are combined with instruct tokens $P_i$ together and fed into the learnable transformers blocks for identity-token disentanglement image generation, enabling good identity consistence and background reconstruction.
• Figure 3: Average cross-attention map of target image in different transformer layers.
• Figure 4: Qualitative comparison on single subject insertion task. Comparison conducted on SpatialSubject200K shows that our method preserves identity consistency and visual coherence in this task, with performance better than recent works.
• Figure 5: Qualitative comparison on multiple subjects insertion task. Comparison conducted on InstructAR Dataset shows that our method preserves identity consistency and visual coherence in this task, with performance better than recent works.