Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Use of Metric Learning for the Recognition of Handwritten Digits, and its Application to Increase the Outreach of Voice-based Communication Platforms

Devesh Pant, Dibyendu Talukder, Deepak Kumar, Rachit Pandey, Aaditeshwar Seth, Chetan Arora

TL;DR

This work tackles digitizing paper-based field records in low-resource settings by building a large handwritten digits dataset from Bihar SHG forms and developing a metric-learning OCR pipeline using triplet loss. The approach uses a homography-guided OCR workflow with a blank-cell detector and compares direct CNN classification to metric-learning embeddings, finding the latter yields superior digit- and phone-level accuracy and better generalization to real-world data. The system is demonstrated in an IVR-based health outreach program, achieving substantial scale (millions of push calls) and providing open-source data, models, and code to enable similar digitization in other contexts. The results highlight the practical viability of combining paper-based data collection with automated digitization to extend digital outreach in communities with limited access to devices and training, and suggest refinements in form design and annotation workflows for improved efficiency.

Abstract

Initiation, monitoring, and evaluation of development programmes can involve field-based data collection about project activities. This data collection through digital devices may not always be feasible though, for reasons such as unaffordability of smartphones and tablets by field-based cadre, or shortfalls in their training and capacity building. Paper-based data collection has been argued to be more appropriate in several contexts, with automated digitization of the paper forms through OCR (Optical Character Recognition) and OMR (Optical Mark Recognition) techniques. We contribute with providing a large dataset of handwritten digits, and deep learning based models and methods built using this data, that are effective in real-world environments. We demonstrate the deployment of these tools in the context of a maternal and child health and nutrition awareness project, which uses IVR (Interactive Voice Response) systems to provide awareness information to rural women SHG (Self Help Group) members in north India. Paper forms were used to collect phone numbers of the SHG members at scale, which were digitized using the OCR tools developed by us, and used to push almost 4 million phone calls. The data, model, and code have been released in the open-source domain.

Use of Metric Learning for the Recognition of Handwritten Digits, and its Application to Increase the Outreach of Voice-based Communication Platforms

TL;DR

This work tackles digitizing paper-based field records in low-resource settings by building a large handwritten digits dataset from Bihar SHG forms and developing a metric-learning OCR pipeline using triplet loss. The approach uses a homography-guided OCR workflow with a blank-cell detector and compares direct CNN classification to metric-learning embeddings, finding the latter yields superior digit- and phone-level accuracy and better generalization to real-world data. The system is demonstrated in an IVR-based health outreach program, achieving substantial scale (millions of push calls) and providing open-source data, models, and code to enable similar digitization in other contexts. The results highlight the practical viability of combining paper-based data collection with automated digitization to extend digital outreach in communities with limited access to devices and training, and suggest refinements in form design and annotation workflows for improved efficiency.

Abstract

Initiation, monitoring, and evaluation of development programmes can involve field-based data collection about project activities. This data collection through digital devices may not always be feasible though, for reasons such as unaffordability of smartphones and tablets by field-based cadre, or shortfalls in their training and capacity building. Paper-based data collection has been argued to be more appropriate in several contexts, with automated digitization of the paper forms through OCR (Optical Character Recognition) and OMR (Optical Mark Recognition) techniques. We contribute with providing a large dataset of handwritten digits, and deep learning based models and methods built using this data, that are effective in real-world environments. We demonstrate the deployment of these tools in the context of a maternal and child health and nutrition awareness project, which uses IVR (Interactive Voice Response) systems to provide awareness information to rural women SHG (Self Help Group) members in north India. Paper forms were used to collect phone numbers of the SHG members at scale, which were digitized using the OCR tools developed by us, and used to push almost 4 million phone calls. The data, model, and code have been released in the open-source domain.
Paper Structure (12 sections, 2 equations, 7 figures, 3 tables)

This paper contains 12 sections, 2 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology and Results
  4. Homography
  5. Blank classification of digit cells
  6. OCR: Direct classification model
  7. OCR: Robust embedding using Metric Learning and Triplet loss
  8. Benchmarking against commercial OCR systems
  9. Segregation for manual annotation
  10. Further observations on form design
  11. Project experience
  12. Conclusions

Figures (7)

  • Figure 1: OCR pipeline
  • Figure 2: The digits in the MNIST and EMNIST datasets are centred, size corrected, and with no noise; however, our dataset has border noise and several digits are partially cropped due to homography inaccuracy.
  • Figure 3: Form variants used in the field
  • Figure 4: (a) Several incorrect key-points are matched, (b) Several incorrect mismatches are eliminated by assuming (approximate) vertical alignment in the scanning process
  • Figure 5: Example of digits misclassified by the direct classification model
  • ...and 2 more figures