Can AI Models Appreciate Document Aesthetics? An Exploration of Legibility and Layout Quality in Relation to Prediction Confidence

TL;DR

This work formulated hypotheses concerning AI behavior in document understanding tasks, specifically anchored in document design principles, and tested four aspects of aesthetic effects: noise, font-size contrast, alignment, and complexity, on model confidence using correlational analysis.

Abstract

A well-designed document communicates not only through its words but also through its visual eloquence. Authors utilize aesthetic elements such as colors, fonts, graphics, and layouts to shape the perception of information. Thoughtful document design, informed by psychological insights, enhances both the visual appeal and the comprehension of the content. While state-of-the-art document AI models demonstrate the benefits of incorporating layout and image data, it remains unclear whether the nuances of document aesthetics are effectively captured. To bridge the gap between human cognition and AI interpretation of aesthetic elements, we formulated hypotheses concerning AI behavior in document understanding tasks, specifically anchored in document design principles. With a focus on legibility and layout quality, we tested four aspects of aesthetic effects: noise, font-size contrast, alignment, and complexity, on model confidence using correlational analysis. The results and observations highlight the value of model analysis rooted in document design theories. Our work serves as a trailhead for further studies and we advocate for continued research in this topic to deepen our understanding of how AI interprets document aesthetics.

Figures (2)

• Figure 1: FUNSD: Examples. On the left, line-level elements and their alignment groups, identified by Algorithm \ref{['alg:alignment']}, are marked in different colors; elements in the orange group at the top align by their center reference points, while elements in the remaining groups align by their left reference points. In the middle, it is a case where an element exhibits excessive contrast, marked in orange boxes. The nearest element of the address line is the company name, emphasized with a significantly larger font size, resulting in a high contrast score as calculated by the formula in Section \ref{['sec:measure-contrast']}. On the right, a case of high contrast due to OCR errors is depicted, where two lines can be mistakenly recognized as one element, such as "IND/LOR" combined with "VOLUME", highlighted in orange. This demonstrates real-world instances where OCR inaccuracies lead to confusion in models' understanding of layout.
• Figure 2: IDL: Examples. One the left, it is an instance of "white" file folders, which in general have lower noise scores as measured by the method described in Section \ref{['sec:measure-noise']}. This lower score is attributed to their large areas of uniform color (i.e., white pixels) and minimal abrupt changes. In the middle, an advertisement document is shown, characterized by rich graphic elements and irregular text arrangement, which typically results in higher noise scores. On the right, On the right, a form with high complexity is displayed, assessed by Bonsiepe's formula bonsiepe1968method, but it also exhibits a higher quality of alignment as evaluated by Algorithm \ref{['alg:alignment']}.