Enhancing Computer Vision with Knowledge: a Rummikub Case Study

TL;DR

An approach to expand the network with explicit knowledge and a separate reasoning component is evaluated, applied to the solving of the popular board game Rummikub, and it is demonstrated that the added background knowledge is equally valuable as two-thirds of the data set.

Abstract

Artificial Neural Networks excel at identifying individual components in an image. However, out-of-the-box, they do not manage to correctly integrate and interpret these components as a whole. One way to alleviate this weakness is to expand the network with explicit knowledge and a separate reasoning component. In this paper, we evaluate an approach to this end, applied to the solving of the popular board game Rummikub. We demonstrate that, for this particular example, the added background knowledge is equally valuable as two-thirds of the data set, and allows to bring down the training time to half the original time.

Figures (3)

• Figure 1: Rummikub set examples
• Figure 2: Overview of the tile detection/classification pipeline
• Figure 3: Ablation experiment results. The dashed red lines indicate the highest accuracy reached by the ANN-only approach.