ByteShield: Adversarially Robust End-to-End Malware Detection through Byte Masking

TL;DR

ByteShield introduces a deterministic byte-masking defense for end-to-end malware detectors that masks input regions with a sliding window, generates multiple masked versions, and uses a threshold-based voting rule to classify. By ensuring adversarial payloads are occluded in at least one masked version, it achieves strong robustness against functionality-preserving attacks while maintaining high accuracy on clean data and offering faster inference than smoothing-based defenses. Across EMBER and BODMAS benchmarks, ByteShield outperforms randomized and (de)randomized smoothing defenses, demonstrates resilience to diverse payload-injection attacks, and shows solid temporal robustness with manageable computational costs. The approach hinges on robust masked-training and a conservative voting strategy to mitigate false positives and maintain reliable malware detection under adversarial pressure.

Abstract

Research has proven that end-to-end malware detectors are vulnerable to adversarial attacks. In response, the research community has proposed defenses based on randomized and (de)randomized smoothing. However, these techniques remain susceptible to attacks that insert large adversarial payloads. To address these limitations, we propose a novel defense mechanism designed to harden end-to-end malware detectors by leveraging masking at the byte level. This mechanism operates by generating multiple masked versions of the input file, independently classifying each version, and then applying a threshold-based voting mechanism to produce the final classification. Key to this defense is a deterministic masking strategy that systematically strides a mask across the entire input file. Unlike randomized smoothing defenses, which randomly mask or delete bytes, this structured approach ensures coverage of the file over successive versions. In the best-case scenario, this strategy fully occludes the adversarial payload, effectively neutralizing its influence on the model's decision. In the worst-case scenario, it partially occludes the adversarial payload, reducing its impact on the model's predictions. By occluding the adversarial payload in one or more masked versions, this defense ensures that some input versions remain representative of the file's original intent, allowing the voting mechanism to suppress the influence of the adversarial payload. Results achieved on the EMBER and BODMAS datasets demonstrate the suitability of our defense, outperforming randomized and (de)randomized smoothing defenses against adversarial examples generated with a wide range of functionality-preserving manipulations while maintaining high accuracy on clean examples.

Figures (11)

• Figure 1: Illustration of the masking process. In the top part, an adversarial payload (highlighted in orange) is injected within a malicious executable file. When this adversarial example is processed by the malware detector, the adversarial payload causes the detector to incorrectly classify the malicious executable as benign. In the bottom part, a mask (highlighted in blue) covers the region containing the adversarial payload. By masking the adversarial payload, its influence on the detector is neutralized, thereby correctly classifying the executable as malicious.
• Figure 2: Illustration of a PE file structure and regions targeted by the adversarial attacks.
• Figure 3: Illustration of our byte masking scheme using a stride equals to 10% and a mask size equals to 50% (blue squares) of the original file's size. Given an input example (containing an adversarial payload represented with the orange square), our defense sequentially strides the mask through different locations, and independently classifies each masked version with the base classifier. Finally, the threshold-based voting mechanism determines the final classification by aggregating the individual predictions for each masked version of the input example.
• Figure 4: Graphical depiction of MalConv's architecture.
• Figure 5: True Positive Rate (TPR) vs. False Positive Rate (FPR) for different values of $M \in \{10, 20, 30, 40, 50\}$. Each subplot corresponds to a specific $M$ value, illustrating how the detection performance varies with the parameters $T$ and $S$. Colors represent different $S$ values, while marker styles indicate the corresponding $T$ values.

Theorems & Definitions (2)

• Remark 1
• Remark 2