Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeepSeek-OCR 2: Visual Causal Flow

Haoran Wei, Yaofeng Sun, Yukun Li

TL;DR

DeepSeek-OCR 2 rethinks vision-language encoding by introducing DeepEncoder V2, an LLM-style vision encoder that performs causal reordering of visual tokens through learnable queries. The encoder combines bidirectional attention for visual tokens with a parallel causal flow mechanism, enabling a two-stage cascade where the encoder reorders 2D content before an autoregressive LLM decoder interprets it. Key contributions include a novel attention mask design, a token-budgeted visual-to-language pathway, and a three-stage training pipeline that preserves visual compression while delivering substantial gains on document-reading benchmarks such as OmniDocBench. The approach points toward native multimodality and genuine 2D reasoning by integrating an optimized, shared encoder with LLM-imposed causal structure, offering practical improvements for OCR pipelines and a foundation for broader omni-modal models.

Abstract

We present DeepSeek-OCR 2 to investigate the feasibility of a novel encoder-DeepEncoder V2-capable of dynamically reordering visual tokens upon image semantics. Conventional vision-language models (VLMs) invariably process visual tokens in a rigid raster-scan order (top-left to bottom-right) with fixed positional encoding when fed into LLMs. However, this contradicts human visual perception, which follows flexible yet semantically coherent scanning patterns driven by inherent logical structures. Particularly for images with complex layouts, human vision exhibits causally-informed sequential processing. Inspired by this cognitive mechanism, DeepEncoder V2 is designed to endow the encoder with causal reasoning capabilities, enabling it to intelligently reorder visual tokens prior to LLM-based content interpretation. This work explores a novel paradigm: whether 2D image understanding can be effectively achieved through two-cascaded 1D causal reasoning structures, thereby offering a new architectural approach with the potential to achieve genuine 2D reasoning. Codes and model weights are publicly accessible at http://github.com/deepseek-ai/DeepSeek-OCR-2.

DeepSeek-OCR 2: Visual Causal Flow

TL;DR

DeepSeek-OCR 2 rethinks vision-language encoding by introducing DeepEncoder V2, an LLM-style vision encoder that performs causal reordering of visual tokens through learnable queries. The encoder combines bidirectional attention for visual tokens with a parallel causal flow mechanism, enabling a two-stage cascade where the encoder reorders 2D content before an autoregressive LLM decoder interprets it. Key contributions include a novel attention mask design, a token-budgeted visual-to-language pathway, and a three-stage training pipeline that preserves visual compression while delivering substantial gains on document-reading benchmarks such as OmniDocBench. The approach points toward native multimodality and genuine 2D reasoning by integrating an optimized, shared encoder with LLM-imposed causal structure, offering practical improvements for OCR pipelines and a foundation for broader omni-modal models.

Abstract

We present DeepSeek-OCR 2 to investigate the feasibility of a novel encoder-DeepEncoder V2-capable of dynamically reordering visual tokens upon image semantics. Conventional vision-language models (VLMs) invariably process visual tokens in a rigid raster-scan order (top-left to bottom-right) with fixed positional encoding when fed into LLMs. However, this contradicts human visual perception, which follows flexible yet semantically coherent scanning patterns driven by inherent logical structures. Particularly for images with complex layouts, human vision exhibits causally-informed sequential processing. Inspired by this cognitive mechanism, DeepEncoder V2 is designed to endow the encoder with causal reasoning capabilities, enabling it to intelligently reorder visual tokens prior to LLM-based content interpretation. This work explores a novel paradigm: whether 2D image understanding can be effectively achieved through two-cascaded 1D causal reasoning structures, thereby offering a new architectural approach with the potential to achieve genuine 2D reasoning. Codes and model weights are publicly accessible at http://github.com/deepseek-ai/DeepSeek-OCR-2.
Paper Structure (27 sections, 2 equations, 5 figures, 4 tables)

This paper contains 27 sections, 2 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Parallelized Queries in Decoder
  4. Parallelized Queries in Projector
  5. LLM-based Multimodal Initialization
  6. Methodology
  7. Architecture
  8. DeepEncoder V2
  9. Vision tokenizer
  10. Language model as vision encoder
  11. Causal flow query
  12. Attention mask
  13. DeepSeek-MoE Decoder
  14. Experimental Settings
  15. Data Engine
  16. ...and 12 more sections

Figures (5)

  • Figure 1: We substitute the CLIP component in DeepEncoder with an LLM-style architecture. By customizing the attention mask, visual tokens utilize bidirectional attention while learnable queries adopt causal attention. Each query token can thus attend to all visual tokens and preceding queries, allowing progressive causal reordering over visual information.
  • Figure 2: This figure shows two computer vision models with parallelized queries: DETR's decoder carion2020end for object detection and BLIP2's Q-former li2023blip for visual token compression. Both employ bidirectional self-attention among queries.
  • Figure 3: DeepSeek-OCR 2 adopts the visual token compression mechanism from DeepEncoder, employing an 80M-parameter image compressor that reduces visual tokens by a factor of 16. DeepEncoder V2 differs by replacing DeepEncoder's CLIP module with a compact language model architecture. Through customized attention masks, this LM-style vision encoder acquires CLIP's knowledge compression capabilities while initiating causal modeling of visual sequences.
  • Figure 4: Token count calculation in DeepEncoder V2. DeepEncoder V2 outputs 256$-$1120 tokens per image using a multi-crop strategy with 0$-$6 local views. With 0 local views, only the global view produces 256 tokens; with 6 local views, the count reaches 1120 (6$\times$144+256).
  • Figure 5: Attention mask architecture of DeepEncoder V2. Concatenation of bidirectional mask (vision tokens, ViT-like) and causal triangular mask (flow tokens, LLM decoder-style).