SkyMemory: A LEO Edge Cache for Transformer Inference Optimization and Scale Out

TL;DR

SkyMemory extends key-value caching to a LEO satellite constellation to reduce inference latency for Transformer-based models by increasing cache hits across an orbital edge. The design combines a baseline KVC protocol with rotation-, hop-, and rotation+hop-aware mappings to place and migrate chunks across a +GRID 2D-Torus network, exploiting inter-satellite laser links. Through simulations and a 5-NUC proof-of-concept, the work demonstrates significant latency reductions and token-generation speedups, validating that migrating caches to the request improves end-to-end performance. The approach generalizes to any distributed cache over multiple locations and offers practical pathways for edge caching in satellite and mobile networks.

Abstract

We expand the scope of cache memory to include LEO constellations, which are highly distributed systems with thousands of satellites connected with free-space optics inter-satellite links (ISL) always only one hop from any point on earth. We show how to increase the number of cache hits and improve the speed of inference for the important use case of LLMs. These benefits apply not only to LLMs, both terrestrially hosted and on satellites, but also generalize to any cache distributed over multiple locations that needs to be accessed in a timely manner. We show the benefit of our key value cache (KVC) protocol in simulations and present a proof-of-concept implementation of the protocol for KVCs on a testbed comprising 5 Intel NUC Linux mini PCs hosting a 19x5 constellation, with an NVIDIA Jetson Nano 8GB GPU hosting the LLM.

Figures (16)

• Figure 1: Dependence of the intra-plane latency on $M$ and $h$ in a three-dimensional plot of latency vs constellation altitude $h$ for different fixed values of $M$, the number of satellites in a single plane of the LEO constellation.
• Figure 2: Dependence of the intra-plane latency on $M$ and $h$ in a contour plot of latency vs constellation altitude $h$ for different fixed values of $M$, the number of satellites in a single plane of the LEO constellation.
• Figure 3: The +GRID networking model, so named because the 4 ISL links from each satellite look like a plus sign. Note that the topology is a torus, so the topmost and bottom-most rows wrap around, as do the leftmost and rightmost columns. Source: Pfandzelter and Bermbach pfandzelter2022.
• Figure 4: Each square of the grid represents one satellite. The squares colored orange are in the LOS from the ground-hosted LLM, while the square representing the satellite closest to the LLM is circled in green. The numbers in the squares represent the logical chunk indices on the servers in the satellites.
• Figure 5: Illustrating chunk migration from the rightmost of the colored columns to the leftmost colored column, to account for rotation of the satellites in Figure \ref{['rotationbefore']}.