Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments

Regan Bolton, Mohammadreza Sheikhfathollahi, Simon Parkinson, Vanessa Vulovic, Gary Bamford, Dan Basher, Howard Parkinson

TL;DR

This work introduces Document Retrieval-Augmented Fine-Tuning (DRAFT), a dual-retrieval, fine-tuning framework designed to improve safety-critical software compliance assessment by jointly querying both documentation and applicable standards. Building on RAFT, DRAFT uses a dual retriever, a distractor-aware fine-tuning dataset, and LoRA-based optimization to enhance evidence-based reasoning and traceability. In experiments with GPT-4o-mini, the approach yields a 7% improvement in correctness over a baseline, with notable qualitative gains in evidence handling, answer structure, and domain-specific reasoning. Although gains are modest, the method offers a practical pathway to more reliable, transparent regulatory assessments in high-stakes domains, particularly when resources or model capability necessitate specialized tuning.

Abstract

Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains.

Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments

TL;DR

This work introduces Document Retrieval-Augmented Fine-Tuning (DRAFT), a dual-retrieval, fine-tuning framework designed to improve safety-critical software compliance assessment by jointly querying both documentation and applicable standards. Building on RAFT, DRAFT uses a dual retriever, a distractor-aware fine-tuning dataset, and LoRA-based optimization to enhance evidence-based reasoning and traceability. In experiments with GPT-4o-mini, the approach yields a 7% improvement in correctness over a baseline, with notable qualitative gains in evidence handling, answer structure, and domain-specific reasoning. Although gains are modest, the method offers a practical pathway to more reliable, transparent regulatory assessments in high-stakes domains, particularly when resources or model capability necessitate specialized tuning.

Abstract

Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains.
Paper Structure (15 sections, 9 equations, 3 figures, 2 tables)

This paper contains 15 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Application context
  2. Related work
  3. Methodology
  4. RAG compliance pipeline
  5. Fine-tuning data
  6. Linking document chunks and compliance questions
  7. Grouping the dataset
  8. Generating answers
  9. Adding in distractors
  10. Performing fine tuning
  11. Experiment and Results
  12. Experiment details
  13. Results
  14. Discussion
  15. Conclusion

Figures (3)

  • Figure 1: Flowchart of the compliance assessment pipeline
  • Figure 2: Prompt template used to generate an answer in the fine tuning dataset
  • Figure 3: 4o-mini model: Full visualization of training and validation loss across all 685 records. The blue line shows the training loss (sampled every 5 points for clarity), while the red line with markers shows validation loss measurements taken at every 10th record. Note the significant decline in both losses during the first 100 records and the stabilization after approximately record 300.