Prompt-Based Caption Generation for Single-Tooth Dental Images Using Vision-Language Models

TL;DR

The findings suggest that guided prompts help VLMs generate meaningful captions and show that the prompts generated by the framework are better anchored in describing the visual aspects of dental images.

Abstract

Digital dentistry has made significant advances with the advent of deep learning. However, the majority of these deep learning-based dental image analysis models focus on very specific tasks such as tooth segmentation, tooth detection, cavity detection, and gingivitis classification. There is a lack of a specialized model that has holistic knowledge of teeth and can perform dental image analysis tasks based on that knowledge. Datasets of dental images with captions can help build such a model. To the best of our knowledge, existing dental image datasets with captions are few in number and limited in scope. In many of these datasets, the captions describe the entire mouth, while the images are limited to the anterior view. As a result, posterior teeth such as molars are not clearly visible, limiting the usefulness of the captions for training vision-language models. Additionally, the captions focus only on a specific disease (gingivitis) and do not provide a holistic assessment of each tooth. Moreover, tooth disease scores are typically assigned to individual teeth, and each tooth is treated as a separate entity in orthodontic procedures. Therefore, it is important to have captions for single-tooth images. As far as we know, no such dataset of single-tooth images with dental captions exists. In this work, we aim to bridge that gap by assessing the possibility of generating captions for dental images using Vision-Language Models (VLMs) and evaluating the extent and quality of those captions. Our findings suggest that guided prompts help VLMs generate meaningful captions. We show that the prompts generated by our framework are better anchored in describing the visual aspects of dental images. We selected RGB images as they have greater potential in consumer scenarios.

Figures (7)

• Figure 1: Example of captions generated using Vision Language Model GPT-4o. These captions include tooth surface, number, and condition. The short caption is written in green and the long caption is written in violet.
• Figure 2: Publicly available dental images are collected and categorized by view. They are processed with a tooth detector, filtered, captioned using prompt-guided vision-language models aligned with dental ontologies, and finally evaluated.
• Figure 3: Examples of diverse tooth images curated from publicly available repositories.
• Figure 4: Detailed flowchart of the proposed framework. This diagram outlines each step in the pipeline — from dataset collection and filtering, to tooth detection, prompt-based caption generation, and evaluation. The two-step prompt engineering strategy is a key component of the caption generation process.
• Figure 5: Illustration of our two-step prompt engineering strategy (left), including basic and refined prompts, and a word cloud showing the most frequent clinical keywords extracted from the generated captions (right).