Metadata Augmentation using NLP, Machine Learning and AI chatbots: A comparison

TL;DR

This study evaluates metadata augmentation for academic libraries by comparing traditional NLP/ML approaches (TF-IDF + XGBoost, BERT-base fine-tuning) with multiple AI-chatbot strategies across several commercial LLMs. Framed as a multiclass college-labeling task using a real-world CMU theses corpus, it shows that chatbots can achieve higher accuracy than limited-data ML baselines but suffer from reliability issues like line-count errors and output inconsistency. The authors propose a pragmatic workflow that blends a fine-tuned BERT model with chatbot-assisted prompts to balance accuracy, control, and usability, and they highlight the need for standardized documentation and API-based deployment to improve reproducibility. Overall, the work provides actionable guidance for librarians and data curators on integrating AI tools for metadata augmentation while outlining critical challenges in reliability and interpretability of LLM-driven workflows.

Abstract

Recent advances in machine learning and artificial intelligence have provided more alternatives for the implementation of repetitive or monotonous tasks. However, the development of AI tools has not been straightforward, and use case exploration and workflow integration are still ongoing challenges. In this work, we present a detailed qualitative analysis of the performance and user experience of popular commercial AI chatbots when used for document classification with limited data. We report the results for a real-world example of metadata augmentation in academic libraries environment. We compare the results of AI chatbots with other machine learning and natural language processing methods such as XGBoost and BERT-based fine tuning, and share insights from our experience. We found that AI chatbots perform similarly among them while outperforming the machine learning methods we tested, showing their advantage when the method relies on local data for training. We also found that while working with AI chatbots is easier than with code, getting useful results from them still represents a challenge for the user. Furthermore, we encountered alarming conceptual errors in the output of some chatbots, such as not being able to count the number of lines of our inputs and explaining the mistake as ``human error''. Although this is not complete evidence that AI chatbots can be effectively used for metadata classification, we believe that the information provided in this work can be useful to librarians and data curators in developing pathways for the integration and use of AI tools for data curation or metadata augmentation tasks.

Figures (14)

• Figure 1: Number of theses and dissertations hosted on both platforms distributed by date of publication.
• Figure 2: Number of theses by College hosted on KiltHub. Colleges are: School of Computer Sciences (SCS), Dietrich College of Humanities and Social Sciences (DCHSC), Tepper School of Business, College of Fine Arts (CFA), College of Engineering (CE) and Mellon College of Science (MCS). Heinz College of Information Systems and Public Policy is not included in the plot since it only had 13 theses.
• Figure 3: Workflow diagram for the three different methods implemented.
• Figure 4: Accuracy results for the models tested. Only the XGBoost and BERT fine-tuned methods were tested with train/test data given the fact that chatbots don't require training data in zero shot scenarios. Error bars correspond to the standard variation of the accuracy for the group of documents (5x70)
• Figure D1: Model performance comparison for all the models tested during the study.