Procedural Knowledge Extraction from Industrial Troubleshooting Guides Using Vision Language Models
Guillermo Gil de Avalle, Laura Maruster, Christos Emmanouilidis
TL;DR
This paper tackles the problem of extracting procedural knowledge (PK) from flowchart-like industrial troubleshooting diagrams to support operator assistance systems. It evaluates two open-weight Vision-Language Models (Pixtral-12B and Qwen2-VL-7B) across standard and augmented prompts using a fixed PK extraction schema, on a proprietary Dutch dataset of 12 guides. The results show entity F1 scores around $0.24$ to $0.34$ and relation F1 scores below $0.11$, with model-specific failure modes such as infinite-loop hallucinations in Qwen and general capacity limits in Pixtral, indicating that autonomous deployment remains challenging. The findings highlight the need for human-in-the-loop workflows, domain-specific fine-tuning, and architectural advances in spatial reasoning or constrained decoding to move toward reliable industrial PK extraction.
Abstract
Industrial troubleshooting guides encode diagnostic procedures in flowchart-like diagrams where spatial layout and technical language jointly convey meaning. To integrate this knowledge into operator support systems, which assist shop-floor personnel in diagnosing and resolving equipment issues, the information must first be extracted and structured for machine interpretation. However, when performed manually, this extraction is labor-intensive and error-prone. Vision Language Models offer potential to automate this process by jointly interpreting visual and textual meaning, yet their performance on such guides remains underexplored. This paper evaluates two VLMs on extracting structured knowledge, comparing two prompting strategies: standard instruction-guided versus an augmented approach that cues troubleshooting layout patterns. Results reveal model-specific trade-offs between layout sensitivity and semantic robustness, informing practical deployment decisions.