Semantic-Aware Contrastive Fine-Tuning: Boosting Multimodal Malware Classification with Discriminative Embeddings

TL;DR

This work tackles semantic overlap in LLM-derived malware descriptions that hinders accurate family classification. It introduces similarity-based contrastive fine-tuning (CFT) with cosine-similarity hard negatives to produce discriminative embeddings that improve few-shot multimodal malware classification under a MAML framework. By combining anchor/positive generation from multiple LLMs, carefully curated hard negatives, and a knowledge-distilled multimodal classifier, the approach yields substantial accuracy gains on CIC-AndMal-2020 and BODMAS, while producing attribute-aware descriptions that align with expert narratives. The proposed method offers a scalable path to integrate semantic and binary features in cybersecurity tasks, enhancing both automated performance and human interpretability of malware descriptions.

Abstract

The rapid evolution of malware variants requires robust classification methods to enhance cybersecurity. While Large Language Models (LLMs) offer potential for generating malware descriptions to aid family classification, their utility is limited by semantic embedding overlaps and misalignment with binary behavioral features. We propose a contrastive fine-tuning (CFT) method that refines LLM embeddings via targeted selection of hard negative samples based on cosine similarity, enabling LLMs to distinguish between closely related malware families. Our approach combines high-similarity negatives to enhance discriminative power and mid-tier negatives to increase embedding diversity, optimizing both precision and generalization. Evaluated on the CIC-AndMal-2020 and BODMAS datasets, our refined embeddings are integrated into a multimodal classifier within a Model-Agnostic Meta-Learning (MAML) framework on a few-shot setting. Experiments demonstrate significant improvements: our method achieves 63.15% classification accuracy with as few as 20 samples on CIC-AndMal-2020, outperforming baselines by 11--21 percentage points and surpassing prior negative sampling strategies. Ablation studies confirm the superiority of similarity-based selection over random sampling, with gains of 10-23%. Additionally, fine-tuned LLMs generate attribute-aware descriptions that generalize to unseen variants, bridging textual and binary feature gaps. This work advances malware classification by enabling nuanced semantic distinctions and provides a scalable framework for adapting LLMs to cybersecurity challenges.

Figures (5)

• Figure 1: Overview of our similarity-based contrastive fine-tuning framework for malware classification. Initially, embeddings from a pre-trained LLM exhibit significant overlap among malware families, leading to ambiguous descriptions and poor classification. In contrast, in similarity-based contrastive fine-tuning, embeddings become discriminative, clearly separating malware families into distinct clusters. This improved embedding space enables the LLM to generate precise, attribute-specific malware descriptions, substantially enhancing malware classification accuracy.
• Figure 2: Visualization of the pre-trained embedding space of malware descriptions generated by LLaMA-3.1-8B, projected into two dimensions using UMAP. Each color represents a different malware family. The significant overlap and lack of clearly defined clusters demonstrate the pre-trained model's limited capability to semantically distinguish among closely related malware families.
• Figure 3: Distribution of cosine similarity scores between candidate negative samples and ground-truth descriptions for LLaMA-3.2-1B. Hard negatives are selected from the right end of the distribution, exhibiting the highest semantic similarity to the ground-truth. These samples create a more challenging CFT setting by forcing the model to distinguish between highly similar descriptions.
• Figure 4: Comparison of LLaMA-3.1-8B embedding spaces for CIC-AndMal-2020 dataset: pre-trained vs. contrastive fine-tuning with random and similarity-based hard negatives. Each color represents a malware family.
• Figure 5: Comparison of LLaMA-3.1-8B embedding spaces for BODMAS dataset: pre-trained vs. contrastive fine-tuning with random and similarity-based hard negatives. Each color represents a malware family.