LLM-Ref: Enhancing Reference Handling in Technical Writing with Large Language Models

TL;DR

LLM-Ref is presented, a writing assistant tool that aids researchers in writing articles from multiple source documents with enhanced reference synthesis and handling capabilities and achieves a significant increase in Ragas score, a comprehensive metric that provides a holistic view of a RAG system's ability to produce accurate, relevant, and contextually appropriate responses.

Abstract

Large Language Models (LLMs) excel in data synthesis but can be inaccurate in domain-specific tasks, which retrieval-augmented generation (RAG) systems address by leveraging user-provided data. However, RAGs require optimization in both retrieval and generation stages, which can affect output quality. In this paper, we present LLM-Ref, a writing assistant tool that aids researchers in writing articles from multiple source documents with enhanced reference synthesis and handling capabilities. Unlike traditional RAG systems that use chunking and indexing, our tool retrieves and generates content directly from text paragraphs. This method facilitates direct reference extraction from the generated outputs, a feature unique to our tool. Additionally, our tool employs iterative response generation, effectively managing lengthy contexts within the language model's constraints. Compared to baseline RAG-based systems, our approach achieves a $3.25\times$ to $6.26\times$ increase in Ragas score, a comprehensive metric that provides a holistic view of a RAG system's ability to produce accurate, relevant, and contextually appropriate responses. This improvement shows our method enhances the accuracy and contextual relevance of writing assistance tools.

Figures (9)

• Figure 1: Architecture of the proposed LLM-Ref. ① Content Extractor extracts texts and references, preserving the paragraph hierarchy of each article. Each article metadata along with respective paragraph summaries extracted from LLM is stored offline. For a given query, in ② Context Retrieval, relevant paragraphs are extracted and combined with prompts to generate answers. The ③ Iterative Output Synthesizer feeds the combined prompt and context to LLM for output text generation based on context length limit. Finally, the ④ Reference Extractor extracts respective references for output text from relevant paragraphs.
• Figure 2: Fine-grained reference samples generated by LLM-Ref when GPT-3.5 is used as the LLM.
• Figure 3: Prompt to find relevant contexts to a query.
• Figure 4: Prompt used to generate the response based on the context for query.
• Figure 5: Prompt used to integrate new context into existing responses.