Knitting Robots: A Deep Learning Approach for Reverse-Engineering Fabric Patterns

TL;DR

This work tackles reverse knitting by introducing a two-stage deep learning pipeline that first converts real fabric images into front labels and then infers complete, knittable stitch labels. The generation phase uses a Refiner+Img2prog architecture to predict 14 front labels, while the inference phase employs a residual CNN to produce 34 complete labels, enabling both single- and multi-yarn patterns. Across imbalanced data, the model achieves 83.1% F1 in front-label generation and up to 97.0% F1 in complete-label inference, with yarn-type–specific training substantially boosting accuracy. The approach demonstrates a clear path toward fully automated robotic knitting by integrating perception and labeling with spatially aware, CNN-based inference, and it outlines practical directions for scaling to color information, variable inputs, and cross-domain textile processes.

Abstract

Knitting, a cornerstone of textile manufacturing, is uniquely challenging to automate, particularly in terms of converting fabric designs into precise, machine-readable instructions. This research bridges the gap between textile production and robotic automation by proposing a novel deep learning-based pipeline for reverse knitting to integrate vision-based robotic systems into textile manufacturing. The pipeline employs a two-stage architecture, enabling robots to first identify front labels before inferring complete labels, ensuring accurate, scalable pattern generation. By incorporating diverse yarn structures, including single-yarn (sj) and multi-yarn (mj) patterns, this study demonstrates how our system can adapt to varying material complexities. Critical challenges in robotic textile manipulation, such as label imbalance, underrepresented stitch types, and the need for fine-grained control, are addressed by leveraging specialized deep-learning architectures. This work establishes a foundation for fully automated robotic knitting systems, enabling customizable, flexible production processes that integrate perception, planning, and actuation, thereby advancing textile manufacturing through intelligent robotic automation.

Figures (11)

• Figure S1: Reverse knitting.
• Figure S2: Data processing workflow.
• Figure S3: Comparison of transfer image with real image.
• Figure S4: Mapping from front label to complete label. The "No" column represents the numerical identifiers assigned to each stitch type. The "name" column lists the abbreviated names of the stitch types. The "color" column indicates the encoded colors associated with each stitch type, which are used for visualization purposes. The "image" column (left table) is reserved for displaying physical representations or diagrams of the corresponding stitch types. (a) sj map; (b) mj map.
• Figure S5: Physical sheet.