An annotated grain kernel image database for visual quality inspection

TL;DR

A machine vision-based database named GrainSet, which contains more than 350 K single-kernel images with experts’ annotations, will facilitate future research in fields such as assisting inspectors in grain quality inspections, providing guidance for grain storage and trade, and contributing to applications of smart agriculture.

Abstract

We present a machine vision-based database named GrainSet for the purpose of visual quality inspection of grain kernels. The database contains more than 350K single-kernel images with experts' annotations. The grain kernels used in the study consist of four types of cereal grains including wheat, maize, sorghum and rice, and were collected from over 20 regions in 5 countries. The surface information of each kernel is captured by our custom-built device equipped with high-resolution optic sensor units, and corresponding sampling information and annotations include collection location and time, morphology, physical size, weight, and Damage & Unsound grain categories provided by senior inspectors. In addition, we employed a commonly used deep learning model to provide classification results as a benchmark. We believe that our GrainSet will facilitate future research in fields such as assisting inspectors in grain quality inspections, providing guidance for grain storage and trade, and contributing to applications of smart agriculture.

Figures (5)

• Figure 1: Protocol of data collection and annotations for GrainSet. The red superscript of each step corresponds to Figure \ref{['fig:whole-pip']}. a, Raw cereal grains are sampled from trucks, and undergo a series of processes, including using sieving, high-power fan treatment and storage under optimal conditions. Sample information including location, time, species and sub-species is used as annotations. b, During the pre-processing stage, inspectors classify grain samples into eight categories, including normal, six types of DU grains and impurities. Each category including many samples is further divided into many groups based on physical weight. The physical weight and DU grain information are used as annotations. c, In the post-processing stage, inspectors depict kernel shapes and pair UP and DOWN angles. The images, shapes and physical size are used as annotations. d, Annotations from different stages are stored in the .XML file.
• Figure 2: Overview of data collection and annotations for GrainSet.The red superscript of each step corresponds to Figure \ref{['fig:protocol']}. a,GrainSet consists of four common types of cereal grains: wheat, maize, sorghum and rice, and raw cereal grains are collected from over 20 regions in 5 countries. Raw samples undergo a series of processes to remove impurities. b, Inspectors divided grain kernels into predefined batches, with each batch sharing the same DU grain category and physical weight information. c, Our prototype device is built to capture the visual information of each batch, producing a pair of high-resolution UP and DOWN images. d, Inspectors manually depict morphology and align single-kernel images from UP and DOWN angles. All single-kernel images and annotations are combined as GrainSet.
• Figure 3: Examples and distributions of sample information in GrainSet.a, Examples of normal, each kind of DU grains, and impurities for four types of cereal grains. The abbreviations (e.g., NOR) are used in subsequent content. Red lines highlight discriminative regions. b, Percentages of normal, each kind of DU grains, and impurities for four types of cereal grains. c, Percentages and numbers of regional information for four types of cereal grains. d, The inner pie: percentages of four types of cereal grains in GrainSet. The outer pie: percentages of regional information in GrainSet.
• Figure 4: Overview of the model and results of technical validation.a, The detailed structure of our fine-grained classification model. It employs ResNet-50 as the backbone and SE-attention module as the prediction head. b, The performance of our trained models on test sets for four types of cereal grains. The recall, accuracy and F1-score are reported in the confusion matrix c, The t-SNE visualization results of features on test sets for four types of cereal grains.
• Figure 5: The Gard-CAM visualization results extracted from our trained models, and red lines highlight the discriminative regions.