Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Citekit: A Modular Toolkit for Large Language Model Citation Generation

Jiajun Shen, Tong Zhou, Yubo Chen, Kang Liu

TL;DR

This paper addresses the lack of reproducibility and fair, comprehensive evaluation in LLM citation generation. It presents Citekit, an open-source, modular toolkit that decomposes citation pipelines into four modules—Input, Generation Module, Enhancing Module, and Evaluator—with 14 components to enable flexible pipeline construction. Through 11 baseline recipes and the proposed PEEP method, the authors demonstrate how different components contribute to answer accuracy and citation quality, highlighting strengths and remaining challenges such as citation granularity. Empirical results on two SOTA LLMs show that modular enhancements (planning, revisers, and editors) can balance answer correctness with citation fidelity, while also exposing limitations in current evaluation data and benchmarks. The work offers a practical framework to reproduce, compare, and innovate in LLM citation generation, with potential impact on research standardization and verifiability in AI-assisted QA.

Abstract

Enabling Large Language Models (LLMs) to generate citations in Question-Answering (QA) tasks is an emerging paradigm aimed at enhancing the verifiability of their responses when LLMs are utilizing external references to generate an answer. However, there is currently no unified framework to standardize and fairly compare different citation generation methods, leading to difficulties in reproducing different methods and a comprehensive assessment. To cope with the problems above, we introduce \name, an open-source and modular toolkit designed to facilitate the implementation and evaluation of existing citation generation methods, while also fostering the development of new approaches to improve citation quality in LLM outputs. This tool is highly extensible, allowing users to utilize 4 main modules and 14 components to construct a pipeline, evaluating an existing method or innovative designs. Our experiments with two state-of-the-art LLMs and 11 citation generation baselines demonstrate varying strengths of different modules in answer accuracy and citation quality improvement, as well as the challenge of enhancing granularity. Based on our analysis of the effectiveness of components, we propose a new method, self-RAG \snippet, obtaining a balanced answer accuracy and citation quality. Citekit is released at https://github.com/SjJ1017/Citekit.

Citekit: A Modular Toolkit for Large Language Model Citation Generation

TL;DR

This paper addresses the lack of reproducibility and fair, comprehensive evaluation in LLM citation generation. It presents Citekit, an open-source, modular toolkit that decomposes citation pipelines into four modules—Input, Generation Module, Enhancing Module, and Evaluator—with 14 components to enable flexible pipeline construction. Through 11 baseline recipes and the proposed PEEP method, the authors demonstrate how different components contribute to answer accuracy and citation quality, highlighting strengths and remaining challenges such as citation granularity. Empirical results on two SOTA LLMs show that modular enhancements (planning, revisers, and editors) can balance answer correctness with citation fidelity, while also exposing limitations in current evaluation data and benchmarks. The work offers a practical framework to reproduce, compare, and innovate in LLM citation generation, with potential impact on research standardization and verifiability in AI-assisted QA.

Abstract

Enabling Large Language Models (LLMs) to generate citations in Question-Answering (QA) tasks is an emerging paradigm aimed at enhancing the verifiability of their responses when LLMs are utilizing external references to generate an answer. However, there is currently no unified framework to standardize and fairly compare different citation generation methods, leading to difficulties in reproducing different methods and a comprehensive assessment. To cope with the problems above, we introduce \name, an open-source and modular toolkit designed to facilitate the implementation and evaluation of existing citation generation methods, while also fostering the development of new approaches to improve citation quality in LLM outputs. This tool is highly extensible, allowing users to utilize 4 main modules and 14 components to construct a pipeline, evaluating an existing method or innovative designs. Our experiments with two state-of-the-art LLMs and 11 citation generation baselines demonstrate varying strengths of different modules in answer accuracy and citation quality improvement, as well as the challenge of enhancing granularity. Based on our analysis of the effectiveness of components, we propose a new method, self-RAG \snippet, obtaining a balanced answer accuracy and citation quality. Citekit is released at https://github.com/SjJ1017/Citekit.
Paper Structure (29 sections, 14 figures, 2 tables)

This paper contains 29 sections, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. System Design
  3. Input
  4. Generation Module
  5. Enhancing Module
  6. Real Time retriever
  7. Ahead Planner
  8. Quality Feedbacker
  9. Output Editor
  10. Extensibility
  11. Evaluator
  12. Pipeline
  13. Usage
  14. Realization of SOTA method.
  15. Customized pipeline modification.
  16. ...and 14 more sections

Figures (14)

  • Figure 1: Illustration of Citation Task. An answer without citation makes readers confused about the actual timeline, but if citations are included, they can understand how the details in the answer actually make sense.
  • Figure 2: The modular design of Citekit . On the left, we show four main modules in Citekit and how they interact with other modules, as well as some predefined components and their abilities; on the right, we illustrate three baseline implementations in our framework and show the data flow during the running of their pipelines
  • Figure 3: An example to define an Attribute First, then Generate pipeline. An attributer will highlight some spans and cluster them, and each cluster will be used to generate an answer sentence.
  • Figure 4: An example to plug in new modules and recreate a new pipeline.
  • Figure 5: Design of PEEP. We show an example of generating a comprehensive answer for an ASQA question.
  • ...and 9 more figures