Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal Prediction for Natural Language Processing: A Survey

Margarida M. Campos, António Farinhas, Chrysoula Zerva, Mário A. T. Figueiredo, André F. T. Martins

TL;DR

Conformal prediction (CP) offers distribution-free uncertainty quantification with coverage guarantees for NLP tasks under exchangeability. The paper surveys CP fundamentals, extensions beyond exchangeability, and NLP applications spanning text classification, sequence tagging, and natural language generation, highlighting how CP yields calibrated prediction sets and risk-controlled outputs without retraining. It discusses efficiency metrics, calibration tools, and open challenges, including token- and sentence-level generation and fairness considerations, providing a roadmap for future work. Overall, the survey demonstrates CP’s practical impact for uncertainty-aware NLP systems and identifies key directions to advance theory and applications in real-world settings.

Abstract

The rapid proliferation of large language models and natural language processing (NLP) applications creates a crucial need for uncertainty quantification to mitigate risks such as hallucinations and to enhance decision-making reliability in critical applications. Conformal prediction is emerging as a theoretically sound and practically useful framework, combining flexibility with strong statistical guarantees. Its model-agnostic and distribution-free nature makes it particularly promising to address the current shortcomings of NLP systems that stem from the absence of uncertainty quantification. This paper provides a comprehensive survey of conformal prediction techniques, their guarantees, and existing applications in NLP, pointing to directions for future research and open challenges.

Conformal Prediction for Natural Language Processing: A Survey

TL;DR

Conformal prediction (CP) offers distribution-free uncertainty quantification with coverage guarantees for NLP tasks under exchangeability. The paper surveys CP fundamentals, extensions beyond exchangeability, and NLP applications spanning text classification, sequence tagging, and natural language generation, highlighting how CP yields calibrated prediction sets and risk-controlled outputs without retraining. It discusses efficiency metrics, calibration tools, and open challenges, including token- and sentence-level generation and fairness considerations, providing a roadmap for future work. Overall, the survey demonstrates CP’s practical impact for uncertainty-aware NLP systems and identifies key directions to advance theory and applications in real-world settings.

Abstract

The rapid proliferation of large language models and natural language processing (NLP) applications creates a crucial need for uncertainty quantification to mitigate risks such as hallucinations and to enhance decision-making reliability in critical applications. Conformal prediction is emerging as a theoretically sound and practically useful framework, combining flexibility with strong statistical guarantees. Its model-agnostic and distribution-free nature makes it particularly promising to address the current shortcomings of NLP systems that stem from the absence of uncertainty quantification. This paper provides a comprehensive survey of conformal prediction techniques, their guarantees, and existing applications in NLP, pointing to directions for future research and open challenges.
Paper Structure (45 sections, 1 theorem, 10 equations, 2 figures)

This paper contains 45 sections, 1 theorem, 10 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Scope.
  3. What this survey is not about.
  4. Conformal Predictors
  5. Definitions and Ingredients
  6. Procedure
  7. Theoretical Guarantees
  8. Relation to Hypothesis Testing
  9. Efficiency Metrics
  10. Pointwise Metrics
  11. Extending Conformal Prediction
  12. Conditional Conformal Predictors
  13. Beyond Exchangeability
  14. Conformal Risk Control
  15. Other CP Variants
  16. ...and 30 more sections

Key Result

theorem 1

Let $(Z_1,...,Z_n,Z_\mathrm{test})$ be an exchangeable sequence of random variables, where $Z_i = (X_i,Y_i) \in \mathcal{X}\times\mathcal{Y}$, and $\mathcal{C}_\alpha: \mathcal{X}\rightarrow 2^\mathcal{Y}$ a conformal predictor as described in §sec:procedure. Then, $\mathcal{C}_\alpha$ satisfies

Figures (2)

  • Figure 1: Survey roadmap: CP variants and their use in NLP applications with examples in the literature.
  • Figure 2: Example of CP for medical report classification ($K$ possible labels).

Theorems & Definitions (1)

  • theorem 1