Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments

Omidreza Safaei, Sinem Aslan, Sebastiano Vascon, Luca Palmieri, Marina Khoroshiltseva, Marcello Pelillo

TL;DR

A human-in-the-loop (HIL) puzzle-solving framework tailored for real-world cultural heritage reconstruction that combines an automatic relaxation-labeling solver with interactive human guidance, enabling users to iteratively lock verified placements, correct errors, and guide assembly toward semantic and geometric coherence.

Abstract

Reassembling real-world archaeological artifacts from fragmented pieces poses significant challenges due to erosion, missing regions, irregular shapes, and large-scale ambiguity. Traditional jigsaw puzzle solvers, often designed for clean synthetic scenarios, struggle under these conditions, especially when the number of fragments grows into the thousands, as in the RePAIR benchmark. In this paper, we propose a human-in-the-loop (HIL) puzzle solving framework designed to address the complexity and scale of real-world cultural heritage reconstruction. Our approach integrates an automatic relaxation-labeling solver with interactive human guidance, allowing users to iteratively lock verified placements, correct errors, and guide the system toward semantically and geometrically coherent assemblies. We introduce two complementary interaction strategies, Iterative Anchoring and Continuous Interactive Refinement, which support scalable reconstruction across varying levels of ambiguity and puzzle size. Experiments on several RePAIR groups demonstrate that our hybrid approach substantially outperforms both fully automatic and manual baselines in accuracy and efficiency, offering a practical solution for large-scale expert-in-the-loop artifact reassembly.

Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments

TL;DR

A human-in-the-loop (HIL) puzzle-solving framework tailored for real-world cultural heritage reconstruction that combines an automatic relaxation-labeling solver with interactive human guidance, enabling users to iteratively lock verified placements, correct errors, and guide assembly toward semantic and geometric coherence.

Abstract

Reassembling real-world archaeological artifacts from fragmented pieces poses significant challenges due to erosion, missing regions, irregular shapes, and large-scale ambiguity. Traditional jigsaw puzzle solvers, often designed for clean synthetic scenarios, struggle under these conditions, especially when the number of fragments grows into the thousands, as in the RePAIR benchmark. In this paper, we propose a human-in-the-loop (HIL) puzzle solving framework designed to address the complexity and scale of real-world cultural heritage reconstruction. Our approach integrates an automatic relaxation-labeling solver with interactive human guidance, allowing users to iteratively lock verified placements, correct errors, and guide the system toward semantically and geometrically coherent assemblies. We introduce two complementary interaction strategies, Iterative Anchoring and Continuous Interactive Refinement, which support scalable reconstruction across varying levels of ambiguity and puzzle size. Experiments on several RePAIR groups demonstrate that our hybrid approach substantially outperforms both fully automatic and manual baselines in accuracy and efficiency, offering a practical solution for large-scale expert-in-the-loop artifact reassembly.
Paper Structure (11 sections, 11 equations, 2 figures, 1 table)

This paper contains 11 sections, 11 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Interactive Puzzle Assembly Framework
  4. Automatic Solver via Relaxation Labeling
  5. Human-in-the-Loop Guided Puzzle Assembly
  6. Initialization Phase
  7. Interactive Control over Strategy Distributions
  8. Interaction Strategies
  9. Experiments
  10. Performance evaluation
  11. Conclusion

Figures (2)

  • Figure 1: Step-by-step visualization of the HIL-IA process. At each iteration $t$, the current anchor $L(t)$ is expanded with solver proposals. User verification or correction yields the next anchor $L(t{+}1)$. Red circles indicate manual adjustments through fragment repositioning that stabilize and guide consistent assembly.
  • Figure 2: Side-by-side visual comparisons between our Human-in-the-Loop solution and the automatic solver khoroshiltseva2024nash output across three groups from the RePAIR benchmark tsesmelis2024repair. Left: Group 1, Center: Group 3, Right: Group 39. In each pair, the left cluster shows the HIL-IA guided reconstruction, while the right cluster shows the fully automatic result.