Show, don't tell -- Providing Visual Error Feedback for Handwritten Documents
Said Yasin, Torsten Zesch
TL;DR
This paper investigates visual error feedback for handwritten documents by analyzing the end-to-end versus modular OCR pipelines in the context of education. It systematically dissects the feedback pipeline into word detection, ordering, recognition, and feedback generation, and compares PGNet (end-to-end) with modular systems like Tesseract, EasyOCR, and a cloud-based Handprint approach. Across IAM and Imgur5K datasets, the study finds that neither architecture delivers classroom-ready performance in detection, ordering, and extraction, highlighting issues such as IoU-dependent feedback placement and misalignment in word ordering and recognition that can distort feedback. A functional prototype using a document camera demonstrates near real-time feedback under favorable spacing, but the authors emphasize the need for character-level BBs, better handling of spacing constraints, and AR-based feedback to translate these insights into practical educational tools. The work lays out a concrete research agenda for improving visual feedback on handwriting and provides public code to catalyze further development.
Abstract
Handwriting remains an essential skill, particularly in education. Therefore, providing visual feedback on handwritten documents is an important but understudied area. We outline the many challenges when going from an image of handwritten input to correctly placed informative error feedback. We empirically compare modular and end-to-end systems and find that both approaches currently do not achieve acceptable overall quality. We identify the major challenges and outline an agenda for future research.