Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Question: How do Large Language Models perform on the Question Answering tasks? Answer:

Kevin Fischer, Darren Fürst, Sebastian Steindl, Jakob Lindner, Ulrich Schäfer

TL;DR

The paper benchmarks QA performance of smaller fine-tuned models against out-of-the-box instruction-following LLMs on SQuAD2, introducing a single-inference prompting strategy to handle unanswerable questions and assess generalization to other QA datasets. It combines an empirical comparison across GPT-4 Turbo, LLaMA variants, and FT-LMs (Flan-T5, DistilBERT, RoBERTa), with extended evaluation using Levenshtein Distance and interrogative-pronoun analyses. Results show that fine-tuned models dominate on in-distribution SQuAD2, while large LLMs close or surpass this gap on out-of-distribution QA tasks, with LLaMA-3.1-70B Instruct delivering particularly strong OOD performance at the cost of higher inference resources. The work highlights the trade-offs between prompting-driven zero-shot/few-shot QA and task-specific fine-tuning, and points to future directions in prompt design for unanswerable questions and evaluating even larger LLMs.

Abstract

Large Language Models (LLMs) have been showing promising results for various NLP-tasks without the explicit need to be trained for these tasks by using few-shot or zero-shot prompting techniques. A common NLP-task is question-answering (QA). In this study, we propose a comprehensive performance comparison between smaller fine-tuned models and out-of-the-box instruction-following LLMs on the Stanford Question Answering Dataset 2.0 (SQuAD2), specifically when using a single-inference prompting technique. Since the dataset contains unanswerable questions, previous work used a double inference method. We propose a prompting style which aims to elicit the same ability without the need for double inference, saving compute time and resources. Furthermore, we investigate their generalization capabilities by comparing their performance on similar but different QA datasets, without fine-tuning neither model, emulating real-world uses where the context and questions asked may differ from the original training distribution, for example swapping Wikipedia for news articles. Our results show that smaller, fine-tuned models outperform current State-Of-The-Art (SOTA) LLMs on the fine-tuned task, but recent SOTA models are able to close this gap on the out-of-distribution test and even outperform the fine-tuned models on 3 of the 5 tested QA datasets.

Question: How do Large Language Models perform on the Question Answering tasks? Answer:

TL;DR

The paper benchmarks QA performance of smaller fine-tuned models against out-of-the-box instruction-following LLMs on SQuAD2, introducing a single-inference prompting strategy to handle unanswerable questions and assess generalization to other QA datasets. It combines an empirical comparison across GPT-4 Turbo, LLaMA variants, and FT-LMs (Flan-T5, DistilBERT, RoBERTa), with extended evaluation using Levenshtein Distance and interrogative-pronoun analyses. Results show that fine-tuned models dominate on in-distribution SQuAD2, while large LLMs close or surpass this gap on out-of-distribution QA tasks, with LLaMA-3.1-70B Instruct delivering particularly strong OOD performance at the cost of higher inference resources. The work highlights the trade-offs between prompting-driven zero-shot/few-shot QA and task-specific fine-tuning, and points to future directions in prompt design for unanswerable questions and evaluating even larger LLMs.

Abstract

Large Language Models (LLMs) have been showing promising results for various NLP-tasks without the explicit need to be trained for these tasks by using few-shot or zero-shot prompting techniques. A common NLP-task is question-answering (QA). In this study, we propose a comprehensive performance comparison between smaller fine-tuned models and out-of-the-box instruction-following LLMs on the Stanford Question Answering Dataset 2.0 (SQuAD2), specifically when using a single-inference prompting technique. Since the dataset contains unanswerable questions, previous work used a double inference method. We propose a prompting style which aims to elicit the same ability without the need for double inference, saving compute time and resources. Furthermore, we investigate their generalization capabilities by comparing their performance on similar but different QA datasets, without fine-tuning neither model, emulating real-world uses where the context and questions asked may differ from the original training distribution, for example swapping Wikipedia for news articles. Our results show that smaller, fine-tuned models outperform current State-Of-The-Art (SOTA) LLMs on the fine-tuned task, but recent SOTA models are able to close this gap on the out-of-distribution test and even outperform the fine-tuned models on 3 of the 5 tested QA datasets.

Paper Structure

This paper contains 16 sections, 1 equation, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Method and Experiments
  4. Models
  5. Prompting Strategy
  6. Unanswerable Questions
  7. Evaluation
  8. Evaluation on the Benchmark
  9. Extended Evaluation
  10. Post-Processing for LLM predictions
  11. Out of Distribution Comparisons
  12. Results and Discussion
  13. Limitations
  14. Conclusion and Future Work
  15. Additional Results
  16. ...and 1 more sections

Figures (6)

  • Figure 1: The mean F1 score of the models discerned by interrogative pronouns.
  • Figure 2: Mean EM score if $n$ character changes (cf. Levenshtein Distance) to the answer were allowed.
  • Figure 3: EM-Scores of each model on the OOD-Datasets.
  • Figure 4: F1-Scores of each model on the OOD-Datasets.
  • Figure 5: The prompt used for GPT4-Turbo
  • ...and 1 more figures