Skip TLB flushes for reused pages within mmap's

TL;DR

This work introduces and implements the "fast page recycling"(FPR) feature within the mmap system call, and shows that TLB shootdowns are a significant source of bottlenecks, previously misattributed to other components of the Linux kernel.

Abstract

Memory access efficiency is significantly enhanced by caching recent address translations in the CPUs' Translation Lookaside Buffers (TLBs). However, since the operating system is not aware of which core is using a particular mapping, it flushes TLB entries across all cores where the application runs whenever addresses are unmapped, ensuring security and consistency. These TLB flushes, known as TLB shootdowns, are costly and create a performance and scalability bottleneck. A key contributor to TLB shootdowns is memory-mapped I/O, particularly during mmap-munmap cycles and page cache evictions. Often, the same physical pages are reassigned to the same process post-eviction, presenting an opportunity for the operating system to reduce the frequency of TLB shootdowns. We demonstrate, that by slightly extending the mmap function, TLB shootdowns for these "recycled pages" can be avoided. Therefore we introduce and implement the "fast page recycling" (FPR) feature within the mmap system call. FPR-mmaps maintain security by only triggering TLB shootdowns when a page exits its recycling cycle and is allocated to a different process. To ensure consistency when FPR-mmap pointers are used, we made minor adjustments to virtual memory management to avoid the ABA problem. Unlike previous methods to mitigate shootdown effects, our approach does not require any hardware modifications and operates transparently within the existing Linux virtual memory framework. Our evaluations across a variety of CPU, memory, and storage setups, including persistent memory and Optane SSDs, demonstrate that FPR delivers notable performance gains, with improvements of up to 28% in real-world applications and 92% in micro-benchmarks. Additionally, we show that TLB shootdowns are a significant source of bottlenecks, previously misattributed to other components of the Linux kernel.

Figures (23)

• Figure 1: TLB shootdowns and compute performance for increasing number of compute threads and a single I/O thread.
• Figure 2: Interaction of the TLB and memory.
• Figure 3: Varying impact of TLB shootdowns from thread T1 for T2, mostly in user space, and T3 mostly in kernel space.
• Figure 4: FPR: Recycling pages are reused without shootdowns. They still are managed by the default OS internals.
• Figure 5: Usage of stale TLB entries during mmap-munmap cycles and FPR's virtual address iteration.