Athena: Synergizing Data Prefetching and Off-Chip Prediction via Online Reinforcement Learning

TL;DR

Athena tackles the memory-latency bottleneck by formulating coordination between data prefetchers and off-chip prediction (OCP) as an online reinforcement learning problem. It introduces a lightweight SARSA-based agent with a partitioned Q-value store (QVStore) that observes system-level telemetry, selects coordination actions, and throttles prefetchers using Q-value guidance, all while using a composite reward to separate action effects from workload variability. Through extensive evaluation across 100 memory-intensive workloads and diverse cache designs, Athena consistently outperforms static, heuristic, and prior RL-based coordination policies with only modest storage overhead. The work demonstrates robust gains across single- and multi-core configurations and different OCP/prefetcher combinations, highlighting the practical viability and adaptability of RL-driven coordination in modern memory systems.

Abstract

Prefetching and off-chip prediction are two techniques proposed to hide long memory access latencies in high-performance processors. In this work, we demonstrate that: (1) prefetching and off-chip prediction often provide complementary performance benefits, yet (2) naively combining them often fails to realize their full performance potential, and (3) existing prefetcher control policies leave significant room for performance improvement behind. Our goal is to design a holistic framework that can autonomously learn to coordinate an off-chip predictor with multiple prefetchers employed at various cache levels. To this end, we propose a new technique called Athena, which models the coordination between prefetchers and off-chip predictor (OCP) as a reinforcement learning (RL) problem. Athena acts as the RL agent that observes multiple system-level features (e.g., prefetcher/OCP accuracy, bandwidth usage) over an epoch of program execution, and uses them as state information to select a coordination action (i.e., enabling the prefetcher and/or OCP, and adjusting prefetcher aggressiveness). At the end of every epoch, Athena receives a numerical reward that measures the change in multiple system-level metrics (e.g., number of cycles taken to execute an epoch). Athena uses this reward to autonomously and continuously learn a policy to coordinate prefetchers with OCP. Our extensive evaluation using a diverse set of memory-intensive workloads shows that Athena consistently outperforms prior state-of-the-art coordination policies across a wide range of system configurations with various combinations of underlying prefetchers, OCPs, and main memory bandwidths, while incurring only modest storage overhead. Athena is freely available at https://github.com/CMU-SAFARI/Athena.

Figures (21)

• Figure 1: Performance line graph of a state-of-the-art off-chip predictor (OCP), POPET hermes, and a state-of-the-art prefetcher, Pythia pythia, across 100 workloads.
• Figure 2: Geomean speedup of POPET, Pythia, Naive, and StaticBest combinations across all workloads.
• Figure 3: Fraction of prefetch fills from off-chip main memory that are inaccurate.
• Figure 4: Geomean speedup of Naive, HPAC, MAB, and StaticBest combinations across all workloads.
• Figure 5: High-level overview of Athena as an RL agent.