On the Lack of Robustness of Binary Function Similarity Systems

TL;DR

This work assesses the robustness of eight binary-function similarity models against a black-box, greedy adversarial attack that perturbatively transforms control-flow and instruction content. The study reveals a pronounced performance-robustness trade-off: untargeted attacks achieve very high success across pools and architectures ($\text{wASR}$ up to $\approx$99%), while targeted attacks are substantially less effective but still impactful ($\approx$40–60% depending on setup). It also shows that attacks transfer more readily in the untargeted setting, and that models relying on CFG or instruction-content features differ in susceptibility, with non-ML baselines generally more robust but weaker on clean data. The findings motivate the development of inherently robust function representations and defense strategies beyond standard adversarial training, given the practical risks to malware analysis, vulnerability detection, and legal/compliance workflows.

Abstract

Binary function similarity, which often relies on learning-based algorithms to identify what functions in a pool are most similar to a given query function, is a sought-after topic in different communities, including machine learning, software engineering, and security. Its importance stems from the impact it has in facilitating several crucial tasks, from reverse engineering and malware analysis to automated vulnerability detection. Whereas recent work cast light around performance on this long-studied problem, the research landscape remains largely lackluster in understanding the resiliency of the state-of-the-art machine learning models against adversarial attacks. As security requires to reason about adversaries, in this work we assess the robustness of such models through a simple yet effective black-box greedy attack, which modifies the topology and the content of the control flow of the attacked functions. We demonstrate that this attack is successful in compromising all the models, achieving average attack success rates of 57.06% and 95.81% depending on the problem settings (targeted and untargeted attacks). Our findings are insightful: top performance on clean data does not necessarily relate to top robustness properties, which explicitly highlights performance-robustness trade-offs one should consider when deploying such models, calling for further research.

Figures (8)

• Figure 1: Targeted attack. (a) Initially, the target variants $V = \{f_{v}^{1}, \cdots, f_{v}^{H}\}$ are not among the top-$K$ most similar functions to $f_Q$ in the pool $P$. (b) After the attack, using $f_{adv}$ as query brings all variants in $V$ into the top-$K$.
• Figure 2: Untargeted attack. (a) Initially, $f_Q$ and its variants $V = \{f_{v}^{1}, \cdots, f_{v}^{H}\}$ are among the top-$K$ most similar functions to $f_Q$ in the pool $P$. (b) After the attack, using $f_{adv}$ as query removes $f_Q$ and its variants from the top-$K$.
• Figure 3: Semantics-preserving transformations embodied in the attack.
• Figure 4: wASR while varying the $K$ value and considering $|P| = 128$. The dotted curve represents the average values across the different models.
• Figure 5: wASR while varying the $K$ value and considering $|P| = 512$. The dotted curve represents the average values across the different models.