Demystifying Feature Engineering in Malware Analysis of API Call Sequences

TL;DR

The paper tackles the challenge of feature engineering in malware classification using API call sequences, comparing knowledge-based hand-crafted features with NLP-based automatic features across CNN, LSTM, and Transformer models on the Avast-CTU CAPEv2 dataset. It demonstrates that knowledge-based features generally outperform NLP-based ones, particularly with smaller datasets, with CNN as the consistently strongest model. A key contribution is an in-depth analysis of which features drive decisions, revealing that API names are highly informative while handles and virtual addresses can be brittle and hard to interpret, underscoring the need for robust feature hashing and interpretable representations. The work provides practical guidance on feature design for malware analysis and highlights areas where interpretability and generalization remain challenging under real-world conditions.

Abstract

Machine learning (ML) has been widely used to analyze API call sequences in malware analysis, which typically requires the expertise of domain specialists to extract relevant features from raw data. The extracted features play a critical role in malware analysis. Traditional feature extraction is based on human domain knowledge, while there is a trend of using natural language processing (NLP) for automatic feature extraction. This raises a question: how do we effectively select features for malware analysis based on API call sequences? To answer it, this paper presents a comprehensive study of investigating the impact of feature engineering upon malware classification.We first conducted a comparative performance evaluation under three models, Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Transformer, with respect to knowledge-based and NLP-based feature engineering methods. We observed that models with knowledge-based feature engineering inputs generally outperform those using NLP-based across all metrics, especially under smaller sample sizes. Then we analyzed a complete set of data features from API call sequences, our analysis reveals that models often focus on features such as handles and virtual addresses, which vary across executions and are difficult for human analysts to interpret.

Figures (8)

• Figure 1: Overview of Feature Extraction and Malware Classification.
• Figure 2: Proportional distribution of different data types across the API sequences of all samples in the dataset.
• Figure 3: An example of an API call sequence report.
• Figure 4: Re-splitting the dataset.
• Figure 5: Classification confusion matrices using different processing methods on the raw and supplemented datasets.