Evaluation of LLM Chatbots for OSINT-based Cyber Threat Awareness

TL;DR

This study tackles improving cybersecurity situational awareness by evaluating how well LLM chatbots can perform OSINT-driven binary classification and NER tasks on Twitter data. It benchmarks commercial and open-source chatbots (e.g., GPT-4, GPT4All, and others) against task-specific baselines, revealing strong binary-classification performance ($F_1$ up to 0.94 for GPT-4 and 0.90 for GPT4All) but notable NER limitations. The findings highlight that, while LLM chatbots can support OSINT-based CTI in classifying threats, they do not yet match specialized models for NER, underscoring the need for targeted improvements to integrate ML efficiently into CTI tooling. The work informs researchers and practitioners on where to focus enhancements in model capabilities and integration strategies to reduce development effort in OSINT-based CTI pipelines.

Abstract

Knowledge sharing about emerging threats is crucial in the rapidly advancing field of cybersecurity and forms the foundation of Cyber Threat Intelligence (CTI). In this context, Large Language Models are becoming increasingly significant in the field of cybersecurity, presenting a wide range of opportunities. This study surveys the performance of ChatGPT, GPT4all, Dolly, Stanford Alpaca, Alpaca-LoRA, Falcon, and Vicuna chatbots in binary classification and Named Entity Recognition (NER) tasks performed using Open Source INTelligence (OSINT). We utilize well-established data collected in previous research from Twitter to assess the competitiveness of these chatbots when compared to specialized models trained for those tasks. In binary classification experiments, Chatbot GPT-4 as a commercial model achieved an acceptable F1 score of 0.94, and the open-source GPT4all model achieved an F1 score of 0.90. However, concerning cybersecurity entity recognition, all evaluated chatbots have limitations and are less effective. This study demonstrates the capability of chatbots for OSINT binary classification and shows that they require further improvement in NER to effectively replace specially trained models. Our results shed light on the limitations of the LLM chatbots when compared to specialized models, and can help researchers improve chatbots technology with the objective to reduce the required effort to integrate machine learning in OSINT-based CTI tools.