MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning

TL;DR

MemOCR tackles the memory bottleneck in long-horizon reasoning under context-window limits by moving from linear textual memory to a 2D visual memory whose layout governs information density. It introduces a two-stage lifecycle: memory drafting in the text domain to encode visual salience, and memory reading in the vision domain via a lightweight renderer that outputs a memory image $V_T$, with budget control via visual tokens. Budget-aware reinforcement learning trained on three QA tasks—$ T_{ ext{std}}$, $ T_{ ext{augM}}$, and $ T_{ ext{augQ}}$—learns a single layout policy (via GRPO) that preserves crucial evidence under extreme compression while retaining detail when budget allows. Across HotpotQA, 2WikiMultiHopQA, Natural Questions, and TriviaQA, MemOCR achieves stronger overall QA performance and markedly better robustness at tight budgets, delivering up to an $ extbf{8}$-fold improvement in effective context utilization and highlighting the practical impact of non-uniform information density in memory systems.

Abstract

Long-horizon agentic reasoning necessitates effectively compressing growing interaction histories into a limited context window. Most existing memory systems serialize history as text, where token-level cost is uniform and scales linearly with length, often spending scarce budget on low-value details. To this end, we introduce MemOCR, a multimodal memory agent that improves long-horizon reasoning under tight context budgets by allocating memory space with adaptive information density through visual layout. Concretely, MemOCR maintains a structured rich-text memory (e.g., headings, highlights) and renders it into an image that the agent consults for memory access, visually prioritizing crucial evidence while aggressively compressing auxiliary details. To ensure robustness across varying memory budgets, we train MemOCR with reinforcement learning under budget-aware objectives that expose the agent to diverse compression levels. Across long-context multi-hop and single-hop question-answering benchmarks, MemOCR outperforms strong text-based baselines and achieves more effective context utilization under extreme budgets.

Figures (8)

• Figure 1: Comparison of memory paradigms. (a) Raw History Memory fetches relevant history passages but suffers from noise and redundancy. (b) Textual Summary Memory allows the agent to summarize the history but suffers from uniform information density, where auxiliary details (gray) consume as much token space as crucial information (green). (c) Visual Memory (Ours) allocates memory budget via visual layout to achieve adaptive information density.
• Figure 2: Framework of MemOCR. (a) Memory Drafting (Text Domain): The LLM agent incrementally updates a rich-text memory based on new incoming chunks, assigning visual priority via formatting and structure. (b) Memory Reading (Vision Domain): The rich text is rendered into a 2D memory image, which serves as the agent's sole working context for answering queries. (c) Budget-Aware Training Objectives: We train the agent under varying degrees of memory compression. The drafting ability is updated via aggregated advantages, while the reading ability is updated via separate advantages.
• Figure 3: Design of the budget-aware training objectives. (1) Standard QA uses the unmodified question and memory for global correctness. (2) QA w/ Augmented Memory requires the visibility of crucial evidence even when the visual memory is heavily compressed. (3) QA w/ Augmented Question ensures detailed information is clearly identified with sufficient tokens. The low-budget, high-detail setting (gray area) is excluded as identifying detailed features under severe compression is infeasible.
• Figure 4: Comparison of accuracy and relative performance drop across varying memory budgets (RQ2). MemOCR degrades more gracefully than textual baselines as budgets tighten. Without visual layout, MemOCR's low-budget robustness drops significantly, which suggests that adaptive information density facilitates more efficient memory budget utilization.
• Figure 5: Oracle analysis of layout regions (RQ3). We compare MemOCR with oracle variants that inject ground-truth evidence into either the Crucial or Detailed region of the rendered memory. While both injections improve accuracy, injecting into the crucial region yields larger gains, especially under tight memory budgets.