Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?

Peiyu Li, Xiaobao Huang, Nitesh V. Chawla

TL;DR

CrochetBench presents CrochetPARADE, an executable DSL-based benchmark to push multimodal models from descriptive understanding toward procedural, executable crochet pattern synthesis. By organizing tasks from stitch recognition to NL-to-DSL translation and validating outputs via a compiler- and renderer-enabled pipeline, the study highlights a clear gap between surface-level language grounding and long-horizon symbolic reasoning required for correct, deployable patterns. Across open-source and closed-source models, performance sharply degrades as the evaluation emphasizes executable fidelity (CSR) and global consistency (PER), revealing limitations in state-tracking and 3D-aware synthesis. The work motivates neuro-symbolic, memory-augmented, and domain-specific pretraining approaches, and proposes a structured evaluation framework that could influence future procedural-generation benchmarks and digital fabrication workflows.

Abstract

We present CrochetBench, a benchmark for evaluating the ability of multimodal large language models to perform fine-grained, low-level procedural reasoning in the domain of crochet. Unlike prior benchmarks that focus on high-level description or visual question answering, CrochetBench shifts the emphasis from describing to doing: models are required to recognize stitches, select structurally appropriate instructions, and generate compilable crochet procedures. We adopt the CrochetPARADE DSL as our intermediate representation, enabling structural validation and functional evaluation via execution. The benchmark covers tasks including stitch classification, instruction grounding, and both natural language and image-to-DSL translation. Across all tasks, performance sharply declines as the evaluation shifts from surface-level similarity to executable correctness, exposing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. CrochetBench offers a new lens for assessing procedural competence in multimodal models and highlights the gap between surface-level understanding and executable precision in real-world creative domains. Code is available at https://github.com/Peiyu-Georgia-Li/crochetBench.

CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?

TL;DR

CrochetBench presents CrochetPARADE, an executable DSL-based benchmark to push multimodal models from descriptive understanding toward procedural, executable crochet pattern synthesis. By organizing tasks from stitch recognition to NL-to-DSL translation and validating outputs via a compiler- and renderer-enabled pipeline, the study highlights a clear gap between surface-level language grounding and long-horizon symbolic reasoning required for correct, deployable patterns. Across open-source and closed-source models, performance sharply degrades as the evaluation emphasizes executable fidelity (CSR) and global consistency (PER), revealing limitations in state-tracking and 3D-aware synthesis. The work motivates neuro-symbolic, memory-augmented, and domain-specific pretraining approaches, and proposes a structured evaluation framework that could influence future procedural-generation benchmarks and digital fabrication workflows.

Abstract

We present CrochetBench, a benchmark for evaluating the ability of multimodal large language models to perform fine-grained, low-level procedural reasoning in the domain of crochet. Unlike prior benchmarks that focus on high-level description or visual question answering, CrochetBench shifts the emphasis from describing to doing: models are required to recognize stitches, select structurally appropriate instructions, and generate compilable crochet procedures. We adopt the CrochetPARADE DSL as our intermediate representation, enabling structural validation and functional evaluation via execution. The benchmark covers tasks including stitch classification, instruction grounding, and both natural language and image-to-DSL translation. Across all tasks, performance sharply declines as the evaluation shifts from surface-level similarity to executable correctness, exposing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. CrochetBench offers a new lens for assessing procedural competence in multimodal models and highlights the gap between surface-level understanding and executable precision in real-world creative domains. Code is available at https://github.com/Peiyu-Georgia-Li/crochetBench.

Paper Structure

This paper contains 45 sections, 1 equation, 3 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Dataset Description
  3. Experiment
  4. Benchmark Tasks.
  5. Model Selection.
  6. Task A: Stitch Recognition
  7. Evaluation.
  8. Task B: Instruction Selection
  9. Evaluation.
  10. Task C: Instruction Generation
  11. Evaluation.
  12. Task D: Instruction-to-DSL Translation
  13. Step-Level Translation
  14. Evaluation.
  15. Project-Level Translation
  16. ...and 30 more sections

Figures (3)

  • Figure 1: Distribution of the top-10 most common project types in CrochetBench.
  • Figure 2: Skill level distribution across the CrochetBench dataset. Note that the "Experienced" slice (1.3%) is annotated externally due to its small size.
  • Figure 3: Example of the CrochetBench translation pipeline. (Left) Natural language crochet instructions from the dataset. (Second) Automatically translated into CrochetPARADE DSL, a formal stitch grammar. (Third) Mesh rendering generated from the DSL. (Right) Target crocheted item image provided in the dataset. This pipeline enables direct text-to-image consistency checks, automated validation, and future training of NL $\rightarrow$ DSL models, analogous to text-to-code generation.