Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis

Hudan Studiawan, Frank Breitinger, Mark Scanlon

TL;DR

The paper tackles the lack of standardized evaluation for applying LLMs to digital forensic timeline analysis by proposing a reproducible methodology and a Windows 11 Plaso-based dataset. It defines a metrics-driven evaluation framework using BLEU and ROUGE, specifies four representative tasks (grep-like search, anomaly detection, event summarization, and EDA), and grounds them with task-specific ground truths. Through a ChatGPT case study, it demonstrates that the approach can quantify LLM performance, revealing strong results for targeted searches and notable dependence on context and data formatting for more complex tasks. The work highlights practical limitations (e.g., large timeline handling) and suggests future work on domain-tuned LLMs and privacy-preserving local deployments to enhance reliability and security in forensic practice.

Abstract

Large language models (LLMs) have seen widespread adoption in many domains including digital forensics. While prior research has largely centered on case studies and examples demonstrating how LLMs can assist forensic investigations, deeper explorations remain limited, i.e., a standardized approach for precise performance evaluations is lacking. Inspired by the NIST Computer Forensic Tool Testing Program, this paper proposes a standardized methodology to quantitatively evaluate the application of LLMs for digital forensic tasks, specifically in timeline analysis. The paper describes the components of the methodology, including the dataset, timeline generation, and ground truth development. Additionally, the paper recommends using BLEU and ROUGE metrics for the quantitative evaluation of LLMs through case studies or tasks involving timeline analysis. Experimental results using ChatGPT demonstrate that the proposed methodology can effectively evaluate LLM-based forensic timeline analysis. Finally, we discuss the limitations of applying LLMs to forensic timeline analysis.

Towards a standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis

TL;DR

The paper tackles the lack of standardized evaluation for applying LLMs to digital forensic timeline analysis by proposing a reproducible methodology and a Windows 11 Plaso-based dataset. It defines a metrics-driven evaluation framework using BLEU and ROUGE, specifies four representative tasks (grep-like search, anomaly detection, event summarization, and EDA), and grounds them with task-specific ground truths. Through a ChatGPT case study, it demonstrates that the approach can quantify LLM performance, revealing strong results for targeted searches and notable dependence on context and data formatting for more complex tasks. The work highlights practical limitations (e.g., large timeline handling) and suggests future work on domain-tuned LLMs and privacy-preserving local deployments to enhance reliability and security in forensic practice.

Abstract

Large language models (LLMs) have seen widespread adoption in many domains including digital forensics. While prior research has largely centered on case studies and examples demonstrating how LLMs can assist forensic investigations, deeper explorations remain limited, i.e., a standardized approach for precise performance evaluations is lacking. Inspired by the NIST Computer Forensic Tool Testing Program, this paper proposes a standardized methodology to quantitatively evaluate the application of LLMs for digital forensic tasks, specifically in timeline analysis. The paper describes the components of the methodology, including the dataset, timeline generation, and ground truth development. Additionally, the paper recommends using BLEU and ROUGE metrics for the quantitative evaluation of LLMs through case studies or tasks involving timeline analysis. Experimental results using ChatGPT demonstrate that the proposed methodology can effectively evaluate LLM-based forensic timeline analysis. Finally, we discuss the limitations of applying LLMs to forensic timeline analysis.
Paper Structure (32 sections, 4 equations, 6 figures, 1 table)

This paper contains 32 sections, 4 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Forensic tool testing and validation
  4. Forensic timeline analysis
  5. LLMs for digital forensics
  6. Proposed methodology
  7. Evaluation metrics
  8. BLEU -- Bilingual Evaluation Understudy
  9. ROUGE -- Recall-Oriented Understudy for Gisting Evaluation
  10. Considerations
  11. Common tasks for forensic timeline analysis
  12. Prompts for running grep of specific terms
  13. Prompts for rule-based anomaly detection
  14. Prompts for event summarization
  15. Prompts for exploratory data analysis
  16. ...and 17 more sections

Figures (6)

  • Figure 1: The proposed methodology for quantitative evaluation of LLM-based timeline analysis
  • Figure 2: A conversation sample between an investigator and ChatGPT. Note that the prompt should be accurate following the OpenAI prompt engineering guides
  • Figure 3: Building ground truth for LLM evaluation
  • Figure 4: An example of a ground truth for the event summarization task in JSON format
  • Figure 5: A bar chart generated by ChatGPT in exploratory data analysis task.
  • ...and 1 more figures