LitLLM: A Toolkit for Scientific Literature Review

TL;DR

LitLLM tackles the problem of efficient, fact-grounded literature reviews by combining retrieval and generation. It leverages Retrieval Augmented Generation (RAG) to grounding the related-work section in retrieved papers, using a keyword-based retrieval from abstracts, LLM-based re-ranking, and two generation modes (zero-shot and plan-based) guided by sentence plans. The contributions include a modular pipeline, integration with Semantic Scholar and OpenAlex, and demonstration that plan-based prompts yield more concise outputs while zero-shot prompts provide broader coverage. The approach aims to reduce hallucinations and time spent on literature reviews, with potential to extend to other domains and longer-context LLMs.

Abstract

Conducting literature reviews for scientific papers is essential for understanding research, its limitations, and building on existing work. It is a tedious task which makes an automatic literature review generator appealing. Unfortunately, many existing works that generate such reviews using Large Language Models (LLMs) have significant limitations. They tend to hallucinate-generate non-factual information-and ignore the latest research they have not been trained on. To address these limitations, we propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles, specialized prompting and instructing techniques with the help of LLMs. Our system first initiates a web search to retrieve relevant papers by summarizing user-provided abstracts into keywords using an off-the-shelf LLM. Authors can enhance the search by supplementing it with relevant papers or keywords, contributing to a tailored retrieval process. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Finally, the related work section is generated based on the re-ranked results and the abstract. There is a substantial reduction in time and effort for literature review compared to traditional methods, establishing our toolkit as an efficient alternative. Our project page including the demo and toolkit can be accessed here: https://litllm.github.io

Figures (7)

• Figure 1: LitLLM Interface. Our system works on the Retrieval Augmented Generation (RAG) principle to generate the literature review grounded in retrieved relevant papers. User needs to provide the abstract in the textbox (in purple) and press send to get the generated related work (in red). First, the abstract is summarized into keywords (Section \ref{['sec:retrieval']}), which are used to query a search engine. Retrieved results are re-ranked (in blue) using the Paper Re-Ranking module (Section \ref{['sec:rerank']}), which is then used as context to generate the related work (Section \ref{['sec:generation']}). Users could also provide a sentence plan (in green) according to their preference to generate a concise, readily usable literature review (See Section \ref{['subsec:plan']}).
• Figure 2: Schematic diagram of the modular pipeline used in our system. In the default setup, we summarize the research abstract into a keyword query, which is used to retrieve relevant papers from an academic search engine. We use an LLM-based reranker to select the most relevant paper relative to the provided abstract. Based on the re-ranked results and the user-provided summary of their work, we use an LLM-based generative model to generate the literature review, optionally controlled by a sentence plan.
• Figure 3: Different retrieval strategies as discussed in Section \ref{['sec:retrieval']}
• Figure 4: Prompt used to summarize the research idea by LLM to search an academic engine
• Figure 5: Ranking prompt based on the permutation generation method