Beyond the Hype: Embeddings vs. Prompting for Multiclass Classification Tasks
Marios Kokkodis, Richard Demsyn-Jones, Vijay Raghavan
TL;DR
The paper investigates whether traditional embeddings-based classifiers can surpass LLM prompt-based approaches for multiclass classification on proprietary Thumbtack data. It formalizes the problem as predicting one of $C=95$ categories from text and image inputs and compares zero-shot prompting (GPT-4o mini) with an OpenClip-based embeddings pipeline followed by a softmax classifier. Across text-only, image-only, and text-image inputs, the embeddings approach achieves notably higher accuracy, better probability calibration (lower Brier score), lower latency, and lower cost, with a 49.5% accuracy gain at k=1 compared to prompting. The findings suggest that, for problems leveraging proprietary data, embeddings-based methods can offer a more reliable and economical solution than prompting, guiding practitioners to go beyond hype when selecting predictive models for deployment.
Abstract
Are traditional classification approaches irrelevant in this era of AI hype? We show that there are multiclass classification problems where predictive models holistically outperform LLM prompt-based frameworks. Given text and images from home-service project descriptions provided by Thumbtack customers, we build embeddings-based softmax models that predict the professional category (e.g., handyman, bathroom remodeling) associated with each problem description. We then compare against prompts that ask state-of-the-art LLM models to solve the same problem. We find that the embeddings approach outperforms the best LLM prompts in terms of accuracy, calibration, latency, and financial cost. In particular, the embeddings approach has 49.5\% higher accuracy than the prompting approach, and its superiority is consistent across text-only, image-only, and text-image problem descriptions. Furthermore, it yields well-calibrated probabilities, which we later use as confidence signals to provide contextualized user experience during deployment. On the contrary, prompting scores are overly uninformative. Finally, the embeddings approach is 14 and 81 times faster than prompting in processing images and text respectively, while under realistic deployment assumptions, it can be up to 10 times cheaper. Based on these results, we deployed a variation of the embeddings approach, and through A/B testing we observed performance consistent with our offline analysis. Our study shows that for multiclass classification problems that can leverage proprietary datasets, an embeddings-based approach may yield unequivocally better results. Hence, scientists, practitioners, engineers, and business leaders can use our study to go beyond the hype and consider appropriate predictive models for their classification use cases.