VisKnow: Constructing Visual Knowledge Base for Object Understanding

TL;DR

VisKnow introduces a cascade framework to construct a multi-modal Visual Knowledge Base (VKB) that integrates text and image data with fine-grained region annotations for object understanding. The authors build AnimalKB as a case study, combining encyclopedia-based textual knowledge, large-scale image collections, and part-level region annotations into a structured knowledge graph with 60 relations and multi-modal entities. They demonstrate that VKB enhances zero-shot recognition and fine-grained VQA, and propose TextBench and PartBench as benchmarks to evaluate knowledge reasoning and part segmentation in the animal domain. The work highlights the practicality of automatically constructing visual knowledge bases that enrich visual understanding and enable knowledge-grounded reasoning across tasks and applications.

Abstract

Understanding objects is fundamental to computer vision. Beyond object recognition that provides only a category label as typical output, in-depth object understanding represents a comprehensive perception of an object category, involving its components, appearance characteristics, inter-category relationships, contextual background knowledge, etc. Developing such capability requires sufficient multi-modal data, including visual annotations such as parts, attributes, and co-occurrences for specific tasks, as well as textual knowledge to support high-level tasks like reasoning and question answering. However, these data are generally task-oriented and not systematically organized enough to achieve the expected understanding of object categories. In response, we propose the Visual Knowledge Base that structures multi-modal object knowledge as graphs, and present a construction framework named VisKnow that extracts multi-modal, object-level knowledge for object understanding. This framework integrates enriched aligned text and image-source knowledge with region annotations at both object and part levels through a combination of expert design and large-scale model application. As a specific case study, we construct AnimalKB, a structured animal knowledge base covering 406 animal categories, which contains 22K textual knowledge triplets extracted from encyclopedic documents, 420K images, and corresponding region annotations. A series of experiments showcase how AnimalKB enhances object-level visual tasks such as zero-shot recognition and fine-grained VQA, and serves as challenging benchmarks for knowledge graph completion and part segmentation. Our findings highlight the potential of automatically constructing visual knowledge bases to advance visual understanding and its practical applications. The project page is available at https://vipl-vsu.github.io/VisKnow.

Figures (12)

• Figure 1: The proposed visual knowledge base is developed through a series of cascading steps: 1) designing knowledge schema, 2) annotating seed content, 3) expanding at scale, and 4) validating and iterating. To minimize construction costs, expert involvement is restricted to essential design and validation tasks, while data collection is delegated to crowdsourcing or automated models.
• Figure 2: Multi-modal AnimalKB constructed by the proposed VisKnow framework can be applied in various aspects, including enhancing knowledge-related visual tasks, and providing annotations required for constructing benchmarks.
• Figure 3: Illustration of various types of visual datasets: (a) image-level annotations, such as bounding boxes and scene graphs in Visual Genome krishna2017visual; (b) category-level annotations, such as attribute annotations in AwA2 lampert2009learning; and (c) datasets that provide both image- and category-level annotations, such as the proposed visual knowledge base with its rich, multi-modal semantic information.
• Figure 4: The framework of the proposed VisKnow consists of four stages, in which text-source and image-source knowledge are extracted separately. Given encyclopedia documents, the LLM extracts entities and relations from the text in a structured format under the guidance of pre-defined knowledge types and a few manual annotations. Visual entities are then fed back into the knowledge schema design to refine the part hierarchy, and regional annotations are applied to the seed images accordingly. For image-source knowledge extraction, we utilize and finetune detectors for object and part localization and verify its results with the Vision-Language Models. Finally, the image regions are aligned with text-source knowledge nodes and incorporated into the visual knowledge base.
• Figure 5: Experts design a general hierarchy for the parent category of animals (in blue), and then for each animal, supplement it with the key parts extracted from the textual knowledge (in orange), thus obtaining the part hierarchy.