TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models

TL;DR

This work introduces TrackList, a fine-grained pipeline to analyze how pretraining data shapes open, non-definition-type responses in medical QA. By pairing open-source models with the RefoMed-EN dataset and multiple evaluation metrics (BERTScore, CLS embeddings, and co-occurrence analyses), the authors show that definition-type answers are learned more reliably, while head terms are prone to paraphrasing and tail terms expose memorization gaps. The study also reveals that scaling small open models does not straightforwardly improve performance and that hallucinations do not scale predictably with size. The resulting insights point to the need for richer linguistic-diversity metrics, targeted handling of head vs tail knowledge, and broader public release of datasets and tooling to advance reproducible, interpretable QA research in sensitive domains like medicine.

Abstract

Large Language Models (LLMs) have proven efficient in giving definition-type answers to user input queries. While for humans giving various types of answers, such as examples and paraphrases, is an easy task, LLMs struggle to provide correct answers for other than definition-type queries. In this study, we evaluated this drop in performance using TrackList, a fine-grained linguistic and statistical analysis pipeline to investigate the impact of the pre-training data on LLMs answers to diverse linguistic queries. We also introduce RefoMed-EN, an English dataset consisting of 6170 human-annotated medical terms alongside their corresponding definitions, denominations, exemplifications, explanations, or paraphrases. We studied whether the high frequency of a concept (head) or low frequency (tail) impacts the language model's performance. We evaluated the quality of the LLM's output using syntactic and semantic similarity metrics, statistical correlations and embeddings. Results showed that the LLM's task performance for definition type questions is the highest, while for the exemplification type it is the lowest. Additionally, we showed that for definition-type questions, large language models are prone to paraphrase more on popular and frequent knowledge and less on tail and technical knowledge, especially in the expert texts.

Figures (3)

• Figure 1: Large Language Models tend to generate more hallucinated definition-style outputs even when directly asked to answer an example-style query for tail knowledge terms.
• Figure 2: The pipeline of our method represented in five steps. 1) The zero-shot inference QA task using the medical concepts from the RefoMed-EN dataset. The dataset was divided in subdatasets according to the query type (detailed presentation in section 3.4). 2) We obtained the frequency in terms of number of documents for each RefoMed-EN concept. 3) We calculated the BERTScore between the LMs output and the RefoMed-EN gold standard. 4) These two values were used to compute Pearson correlations. 5) We computer a probability metric between the CLS embedding score between the term and the n-grams of the output, and their frequency in the pre-training corpora.
• Figure 3: Manual analysis of hallucinations on 100 head and tail terms. Best model is OLMo-1b, with the lowest rate of hallucinations (33%).