$\texttt{MemoryRewardBench}$: Benchmarking Reward Models for Long-Term Memory Management in Large Language Models

TL;DR

MemoryRewardBench is the first benchmark dedicated to evaluating reward models’ ability to assess long-term memory management in large language models. It covers long-context reasoning, multi-turn dialogue understanding, and long-form generation across 10 memory-management settings with context lengths from 8K to 128K tokens, using 13 LLMs as RMs. The study finds that newer generations outperform older ones and that open-source RMs are narrowing the gap with proprietary models, though challenges remain in dynamic memory evaluation and long-range dependencies. The benchmark exposes both capabilities and limitations of current RMs, offering practical guidance to improve RM-based memory supervision for memory-centric LLMs. Overall, MemoryRewardBench provides a scalable, systematic framework for advancing memory-aware evaluation and guiding RM and LLM development.

Abstract

Existing works increasingly adopt memory-centric mechanisms to process long contexts in a segment manner, and effective memory management is one of the key capabilities that enables large language models to effectively propagate information across the entire sequence. Therefore, leveraging reward models (RMs) to automatically and reliably evaluate memory quality is critical. In this work, we introduce $\texttt{MemoryRewardBench}$, the first benchmark to systematically study the ability of RMs to evaluate long-term memory management processes. $\texttt{MemoryRewardBench}$ covers both long-context comprehension and long-form generation tasks, featuring 10 distinct settings with different memory management patterns, with context length ranging from 8K to 128K tokens. Evaluations on 13 cutting-edge RMs indicate a diminishing performance gap between open-source and proprietary models, with newer-generation models consistently outperforming their predecessors regardless of parameter count. We further expose the capabilities and fundamental limitations of current RMs in evaluating LLM memory management across diverse settings.

Figures (22)

• Figure 1: Illustration of holistic processing and segmented processing of long input sequence.
• Figure 2: Illustrations of three memory management patterns. From left to right: Sequential pattern, Parallelism pattern, and Mixed pattern. Each pattern depicts both correct and incorrect memory update trajectories. For clarity, context chunks are omitted, and only intermediate memory states are shown. More details are shown in Appendix \ref{['appdix:benchmark_construction']}.
• Figure 3: Performance comparison between Sequential and Parallel memory management patterns on the long-context reasoning and long-form generation tasks.
• Figure 4: Comparison between process-based and outcome-based reward criteria. Chosen-First indicates that the chosen sample is presented before the rejected sample in the input context fed to the RM, and vice versa.
• Figure 5: Performance trends of RMs with increasing constraint density in long-form generation instructions.