Linking Representations with Multimodal Contrastive Learning

TL;DR

CLIPPINGS tackles the problem of linking individuals or firms across historical document sources in the presence of OCR noise and varied textual representations. It uses symmetric vision-language bi-encoders pre-trained with contrastive learning on image-OCR pairs and optimizes a pooled embedding $z_i = \dfrac{f(x_i) + g(t_i)}{2}$ to pull same-class embeddings together across modalities. The approach yields a large accuracy gain over string-matching baselines (e.g., 94.5% versus ~73%) and demonstrates that purely self-supervised multimodal learning can outperform traditional methods. It also shows that multimodal pre-training improves linkage even when only one modality is available at inference, highlighting practical benefits for large-scale historical record linkage.

Abstract

Many applications require linking individuals, firms, or locations across datasets. Most widely used methods, especially in social science, do not employ deep learning, with record linkage commonly approached using string matching techniques. Moreover, existing methods do not exploit the inherently multimodal nature of documents. In historical record linkage applications, documents are typically noisily transcribed by optical character recognition (OCR). Linkage with just OCR'ed texts may fail due to noise, whereas linkage with just image crops may also fail because vision models lack language understanding (e.g., of abbreviations or other different ways of writing firm names). To leverage multimodal learning, this study develops CLIPPINGS (Contrastively LInking Pooled Pre-trained Embeddings). CLIPPINGS aligns symmetric vision and language bi-encoders, through contrastive language-image pre-training on document images and their corresponding OCR'ed texts. It then contrastively learns a metric space where the pooled image-text embedding for a given instance is close to embeddings in the same class (e.g., the same firm or location) and distant from embeddings of a different class. Data are linked by treating linkage as a nearest neighbor retrieval problem with the multimodal embeddings. CLIPPINGS outperforms widely used string matching methods by a wide margin in linking mid-20th century Japanese firms across financial documents. A purely self-supervised model - trained only by aligning the embeddings for the image crop of a firm name and its corresponding OCR'ed text - also outperforms popular string matching methods. Fascinatingly, a multimodally pre-trained vision-only encoder outperforms a unimodally pre-trained vision-only encoder, illustrating the power of multimodal pre-training even if only one modality is available for linking at inference time.

Figures (4)

• Figure 1: This figure shows representative Japanese firm records, taken from a publication on trading partners (left) and a firm index (right). The text orientation of these publications differs. The numbering denotes linked firms. Pink text denotes OCR errors.
• Figure 2: Model Architecture: This figure illustrates the CLIPPINGS model architecture.
• Figure 3: Errors: This figure shows errors in record linkage made by different models.
• Figure 4: Input-Output Networks: This figure plots the average supply chain distance of Japanese firms to Mitsui, Mitsubishi, and Sumitomo, the three most prominent Japanese firms.