Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

TL;DR

Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.

Abstract

LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.

Figures (8)

• Figure 1: Overview of Online Adaptation to Continual Knowledge Streams. At each time interval $t$, a new context chunk $c_t$ is streamed, and the model is queried with context accumulated up to $t$ and a question $q_j$. Performance is calculated as average accuracy across all intervals by comparing predictions with the ground truth answer $a_t$, then averaging over all questions. A Phase $T_i$ denotes a contiguous set of chunks sharing the same ground truth answers. Answer options are limited to OAKS-N, as OAKS-B uses an open-ended format.
• Figure 2: Accuracy (%) across different question types of OAKS-B.
• Figure 3: Accuracy (%) across the timestep where the question is asked.
• Figure 4: Frequency distribution of answer changes per question for OAKS-B and OAKS-N.
• Figure 5: Pass@k performance as the number of context increases for retrieval in RAG on OAKS-B and OAKS-N