Knowledge Prompting: How Knowledge Engineers Use Large Language Models

TL;DR

Knowledge Prompting investigates how knowledge engineers can leverage LLMs to support KG construction and maintenance. The authors conduct a four-day hackathon with ethnographic observations, documents, and interviews to understand challenges, evaluation, skills, and responsible AI considerations. They identify prompting as a critical yet undervalued skill, and propose KG Cards and Model Cards to improve transparency and safety in LLM-enabled KE. The study offers design guidance for KE AI copilots and sets a research agenda for bias mitigation and responsible deployment of LLM-enabled KGs.

Abstract

Despite many advances in knowledge engineering (KE), challenges remain in areas such as engineering knowledge graphs (KGs) at scale, keeping up with evolving domain knowledge, multilingualism, and multimodality. Recently, KE has used LLMs to support semi-automatic tasks, but the most effective use of LLMs to support knowledge engineers across the KE activites is still in its infancy. To explore the vision of LLM copilots for KE and change existing KE practices, we conducted a multimethod study during a KE hackathon. We investigated participants' views on the use of LLMs, the challenges they face, the skills they may need to integrate LLMs into their practices, and how they use LLMs responsibly. We found participants felt LLMs could contribute to improving efficiency when engineering KGs, but presented increased challenges around the already complex issues of evaluating the KE tasks. We discovered prompting to be a useful but undervalued skill for knowledge engineers working with LLMs, and note that natural language processing skills may become more relevant across more roles in KG construction. Integrating LLMs into KE tasks needs to be mindful of potential risks and harms related to responsible AI. Given the limited ethical training, most knowledge engineers receive solutions such as our suggested `KG cards' based on data cards could be a useful guide for KG construction. Our findings can support designers of KE AI copilots, KE researchers, and practitioners using advanced AI to develop trustworthy applications, propose new methodologies for KE and operate new technologies responsibly.

Figures (2)

• Figure 1: The KG lifecycle today, illustrating the stages from requirement collection to deployment. Each phase --- requirement elicitation, KG construction, maintenance, and deployment --- involves collaboration across roles like domain experts, knowledge engineers, and ML engineers to build, enrich, and deploy KGs for intelligent applications.
• Figure 2: Six suggested sections of KG Cards with related details.