Foundations of Unknown-aware Machine Learning
Xuefeng Du
TL;DR
This thesis tackles reliability under distributional uncertainty and unknown classes by formulating unknown-aware learning. It introduces three core strands: (i) VOS and Dream-ood for tractable, interpretable synthetic outliers that regularize decision boundaries and improve OOD detection, (ii) SIREN for shaping object-level representations into compact, class-specific von Mises-Fisher distributions with effective test-time OOD scores, and (iii) SAL and HaloScope for leveraging unlabeled wild data and unlabeled LLM generations to improve OOD detection and hallucination safety. Theoretical analysis provides separability and learnability guarantees for the unlabeled-data approaches, while extensive empirical evaluations demonstrate state-of-the-art performance in OOD detection for object detection and image classification, robust hallucination detection across LLMs, and scalable reliability improvements for foundation models. Together, these results establish unknown-aware learning as a practical paradigm that improves AI safety and reliability with minimal human supervision, spanning vision, language, and multimodal models. The work demonstrates formal reliability guarantees, interpretable outlier generation, and scalable strategies for deploying trustworthy foundation models in the wild.
Abstract
Ensuring the reliability and safety of machine learning models in open-world deployment is a central challenge in AI safety. This thesis develops both algorithmic and theoretical foundations to address key reliability issues arising from distributional uncertainty and unknown classes, from standard neural networks to modern foundation models like large language models (LLMs). Traditional learning paradigms, such as empirical risk minimization (ERM), assume no distribution shift between training and inference, often leading to overconfident predictions on out-of-distribution (OOD) inputs. This thesis introduces novel frameworks that jointly optimize for in-distribution accuracy and reliability to unseen data. A core contribution is the development of an unknown-aware learning framework that enables models to recognize and handle novel inputs without labeled OOD data. We propose new outlier synthesis methods, VOS, NPOS, and DREAM-OOD, to generate informative unknowns during training. Building on this, we present SAL, a theoretical and algorithmic framework that leverages unlabeled in-the-wild data to enhance OOD detection under realistic deployment conditions. These methods demonstrate that abundant unlabeled data can be harnessed to recognize and adapt to unforeseen inputs, providing formal reliability guarantees. The thesis also extends reliable learning to foundation models. We develop HaloScope for hallucination detection in LLMs, MLLMGuard for defending against malicious prompts in multimodal models, and data cleaning methods to denoise human feedback used for better alignment. These tools target failure modes that threaten the safety of large-scale models in deployment. Overall, these contributions promote unknown-aware learning as a new paradigm, and we hope it can advance the reliability of AI systems with minimal human efforts.