How Good Are LLMs at Out-of-Distribution Detection?

TL;DR

The study interrogates how well large language models detect inputs that shift away from training data, focusing on LLaMA models from 7B to 65B. It compares discriminative versus generative fine-tuning, and evaluates both far-OOD and near-OOD scenarios using post-hoc detectors, with a strong emphasis on distance-based measures. A key finding is that simple cosine distance on LLM embeddings achieves superior OOD detection due to the isotropic nature of LLM representations, while generative fine-tuning offers robust, data-efficient improvements over discriminative tuning. The work also demonstrates that zero-shot far-OOD detection is near-perfect with LLMs, that near-OOD detection improves with scale, and that quantization can impact OOD performance. The authors release code to promote reproducibility and further research in reliable OOD detection for decoder-based LLMs in dynamic environments.

Abstract

Out-of-distribution (OOD) detection plays a vital role in enhancing the reliability of machine learning (ML) models. The emergence of large language models (LLMs) has catalyzed a paradigm shift within the ML community, showcasing their exceptional capabilities across diverse natural language processing tasks. While existing research has probed OOD detection with relative small-scale Transformers like BERT, RoBERTa and GPT-2, the stark differences in scales, pre-training objectives, and inference paradigms call into question the applicability of these findings to LLMs. This paper embarks on a pioneering empirical investigation of OOD detection in the domain of LLMs, focusing on LLaMA series ranging from 7B to 65B in size. We thoroughly evaluate commonly-used OOD detectors, scrutinizing their performance in both zero-grad and fine-tuning scenarios. Notably, we alter previous discriminative in-distribution fine-tuning into generative fine-tuning, aligning the pre-training objective of LLMs with downstream tasks. Our findings unveil that a simple cosine distance OOD detector demonstrates superior efficacy, outperforming other OOD detectors. We provide an intriguing explanation for this phenomenon by highlighting the isotropic nature of the embedding spaces of LLMs, which distinctly contrasts with the anisotropic property observed in smaller BERT family models. The new insight enhances our understanding of how LLMs detect OOD data, thereby enhancing their adaptability and reliability in dynamic environments. We have released the source code at \url{https://github.com/Awenbocc/LLM-OOD} for other researchers to reproduce our results.

Figures (6)

• Figure 1: Illustration of two types of OOD instances compared to ID samples: far-OOD where ID and OOD data come from different domains and near-OOD where ID and OOD data come from the same domain but with different classes.
• Figure 2: Our proposed evaluation framework for LLMs at OOD detection, taking three aspects into consideration: ① distribution of OOD samples (near or far), ② impact of model tuning on OOD detection, and ③ diverse OOD score functions.
• Figure 3: UMAP mcinnes2018umap visualization of representations generated from the penultimate layer of the zero-grad () and fine-tuned () LLaMA-7B models. (a) Far-OOD: 20NG is treated as ID while SST-2, RTE, and TREC are treated as OOD. (b) Near-OOD: the banking domain of CLINC150 is selected, of which 50% of the classes are treated as ID, and the rest are treated as OOD.
• Figure 4: Impact of fine-tuning on logits-based OOD scores (MSP at the top row and Energy at the bottom row). We plot SST-2 (ID) vs. TREC-10 (OOD) for visualization.
• Figure 5: Performance changes for ID classification and OOD detection as training progresses with the different number of training shots. Top row: 20NG is ID training task; Bottom row: banking domain of CLINC150 is selected where 50% classes are used as ID training task and the rest are OOD samples. The star ($\star$) on each line indicates the selected results whose epoch corresponds to the best ID performance on the validation set.