Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms

TL;DR

This paper implements and evaluates a solution for the Zoominfo product for sellers, which prompts the LLM with natural language, producing search fields through entity extraction that are then converted into a search query, and manually created the ground truth for 500+ natural language queries.

Abstract

Enterprise searches require users to have complex knowledge of queries, configurations, and metadata, rendering it difficult for them to access information as needed. Most go-to-market (GTM) platforms utilize advanced search, an interface that enables users to filter queries by various fields using categories or keywords, which, historically, however, has proven to be exceedingly cumbersome, as users are faced with seemingly hundreds of options, fields, and buttons. Consequently, querying with natural language has long been ideal, a notion further empowered by Large Language Models (LLMs). In this paper, we implement and evaluate a solution for the Zoominfo product for sellers, which prompts the LLM with natural language, producing search fields through entity extraction that are then converted into a search query. The intermediary search fields offer numerous advantages for each query, including the elimination of syntax errors, simpler ground truths, and an intuitive format for the LLM to interpret. We paired this pipeline with many advanced prompt engineering strategies, featuring an intricate system message, few-shot prompting, chain-of-thought (CoT) reasoning, and execution refinement. Furthermore, we manually created the ground truth for 500+ natural language queries, enabling the supervised fine-tuning of Llama-3-8B-Instruct and the introduction of sophisticated numerical metrics. Comprehensive experiments with closed, open source, and fine-tuned LLM models were conducted through exact, Jaccard, cosine, and semantic similarity on individual search entities to demonstrate the efficacy of our approach. Overall, the most accurate closed model had an average accuracy of 97% per query, with only one field performing under 90%, with comparable results observed from the fine-tuned models.

Figures (5)

• Figure 1: The system design of the search service. The user and user query are on the left side while the search service on the right. The LLM-based conversion with feedback is shown in the middle.
• Figure 2: A screenshot of ZI Sales showing a search for the chief technology officers (CTOs) of companies in the education industry with at least 500 employees and at least $50M revenue. The list has 603 names spread over multiple search results pages. The contact (person) and company names are not shown here due to privacy reasons.
• Figure 3: Score histograms and average (mean) scores for Sonnet 3.5. From left to right: the average score of every metric for an individual query; the average scores for all metrics; the score of every exact match metric.
• Figure 4: The score histograms and average (mean) scores of Sonnet 3.5 vs Haiku vs GPT-3.5-Turbo per metric. Note that each LLM has unique system messages and sets of shots to optimize its own score.
• Figure 5: The score histograms and average (mean) scores for the prompts within the test set. Around 130 queries of synthetic data were also created to test the difference between the models.