BLUEPRINT Rebuilding a Legacy: Multimodal Retrieval for Complex Engineering Drawings and Documents

TL;DR

The paper tackles cross-modal retrieval in legacy engineering archives with inconsistent metadata. It introduces Blueprint, a layout-aware multimodal retrieval framework that routes files to vision or NLP paths, detects layout regions, performs region-restricted OCR, normalizes identifiers, and fuses lexical and dense signals in a unified index, followed by region-aware reranking. On a 5k-file benchmark with 375 domain-specific queries, Blueprint achieves $nDCG@3 = 0.626$ and $Succ@3 = 0.715$, outperforming strong VLM baselines and delivering high throughput ($\approx$9.7 s/file), with robust human and LLM-based evaluation. The work provides a reproducible evaluation protocol, ablation studies quantifying the value of region detection and OCR, and release plans for benchmarks and code to accelerate modernization of legacy engineering archives.

Abstract

Decades of engineering drawings and technical records remain locked in legacy archives with inconsistent or missing metadata, making retrieval difficult and often manual. We present Blueprint, a layout-aware multimodal retrieval system designed for large-scale engineering repositories. Blueprint detects canonical drawing regions, applies region-restricted VLM-based OCR, normalizes identifiers (e.g., DWG, part, facility), and fuses lexical and dense retrieval with a lightweight region-level reranker. Deployed on ~770k unlabeled files, it automatically produces structured metadata suitable for cross-facility search. We evaluate Blueprint on a 5k-file benchmark with 350 expert-curated queries using pooled, graded (0/1/2) relevance judgments. Blueprint delivers a 10.1% absolute gain in Success@3 and an 18.9% relative improvement in nDCG@3 over the strongest vision-language baseline}, consistently outperforming across vision, text, and multimodal intents. Oracle ablations reveal substantial headroom under perfect region detection and OCR. We release all queries, runs, annotations, and code to facilitate reproducible evaluation on legacy engineering archives.

Figures (9)

• Figure 1: Blueprint processes an engineer’s natural-language query and returns multimodal results across drawings, schematics, and procedural documents.
• Figure 2: Representative samples: (a) policy/procedure with structured text, (b) assembly drawing with parts list and annotations, (c) electrical schematic with connection diagrams. Despite facility-specific conventions and missing regions in some drawings, common metadata areas (title block, drawing number, parts list, revisions) enable structured, layout-aware extraction for retrieval.
• Figure 3: Our multimodal retrieval system processes $\sim$770k unlabeled technical documents through parallel NLP and Vision pipelines. Documents are first routed via zero-shot multimodal classification (CLIP + domain-specific CV heuristics). The NLP pipeline performs document filtering and text embedding extraction, while the Vision pipeline applies preprocessing, spatial cropping, and visual encoding. Both modalities populate a unified multimodal vector index with sharded storage for scalability. At query time, encoded queries retrieve candidates from the index and a re-ranking stage returns the top-$K$ most relevant documents, enabling robust search over heterogeneous document types without manual metadata.
• Figure 4: Heatmap showing which baselines win against Blueprint across query intents. “Either” entries reflect queries where judges accept drawings or documents, and values show the percentage of losses per intent.
• Figure 5: Latency–accuracy trade-off across systems. Blueprint attains the fastest end-to-end processing time ($\approx 9.46$ s/file) and the highest retrieval accuracy ($\mathrm{Succ}@3 = 71.5\%$), outperforming all vision–language baselines. Bubble size indicates parameter count.