Do we still need canaries in the coal mine? Measuring shadow stack effectiveness in countering stack smashing

TL;DR

Do we still need canaries in the coal mine? Measuring shadow stack effectiveness in countering stack smashing compares traditional stack canaries with hardware-assisted shadow stacks on x86-64 systems. Using the Juliet C/C++ Test Suite and SPEC CPU 2017 benchmarks across GCC and Clang with multiple optimization settings, the study reveals that compiler-generated stack layouts substantially influence detection rates, with Clang generally outperforming GCC for canaries and shadow stacks not always outperforming canaries. A layout-aware enhancement to Clang's shadow stack improves detection accuracy with only minimal performance impact, suggesting that the most effective protection may arise from combining informed stack layouts with shadow-stack checks. The work highlights the critical role of stack layout and compiler options in memory-safety mitigations and points to practical avenues for elevating defense without prohibitive overhead.

Abstract

Stack canaries and shadow stacks are widely deployed mitigations to memory-safety vulnerabilities. While stack canaries are introduced by the compiler and rely on sentry values placed between variables and control data, shadow stack implementations protect return addresses explicitly and rely on hardware features available in modern processor designs for efficiency. In this paper we hypothesize that stack canaries and shadow stacks provide similar levels of protections against sequential stack-based overflows. Based on the Juliet test suite, we evaluate whether 64-bit x86 (x86-64) systems benefit from enabling stack canaries in addition to the x86-64 shadow stack enforcement. We observe divergence in overflow detection rates between the GCC and Clang compilers and across optimization levels, which we attribute to differences in stack layouts generated by the compilers. We also find that x86-64 shadow stack implementations are more effective and outperform stack canaries when combined with a stack-protector-like stack layout. We implement and evaluate an enhancement to the Clang x86-64 shadow stack instrumentation that improves the shadow stack detection accuracy based on this observation.

Figures (3)

• Figure 1: Stack layout in the architecture (\ref{['fig:nostack-protector']}). A stack canary (\ref{['fig:stack-protector']} ➀) is placed on the stack between the local variables (➁) and saved rbp value (➂) so that the canary will be overwritten in case a stack buffer overflows into the frame record. A shadow stack protects the return address in the frame record (\ref{['fig:fcf-protection']}). If the frame pointer (rbp) is omitted, the protected stack pointer acts as a canary value (\ref{['fig:omit-frame-pointer']}).
• Figure 2: Comparison of Juliet test results by compiler, optimization level, and options. The stack canaries bars show the detection rates for different stack canary options (indicated in the legend) with the shadow stack disabled. The SHSTK bars show the detection rate for the shadow stack separately, and when combined with different stack canary options (indicated in the legend). The SHSTK with canary layout bars show the detection rate for the proof-of-concept options discussed in \ref{['sec:improvedetection']} for Clang -O0 and -O2 configurations. The left axis shows the number of test cases with detections, while the right axis shows the percentage of detected cases relative to the Selected test cases shown in \ref{['tab:cwe_categories']}. Raw values are available in \ref{['tab:juliet-detections-base']} and \ref{['tab:juliet-detections-mod']} in the Appendix.
• Figure 3: Relative performance degradation for SPEC CPU 2017 benchmarks normalized to the baseline with shadow stack and stack canaries disabled. The bars for the -fstack-layout -family of compiler flags show the performance for the proof-of-concept shadow stack instrumentation discussed in \ref{['sec:improvedetection']}.