A Hyperspectral Imaging Guided Robotic Grasping System

TL;DR

This work introduces PRISM, a distortion-free hyperspectral imaging device, and the SpectralGrasp framework, which combine spectral and spatial information to guide robotic grasping and textile sorting. By reconstructing and correcting hyperspectral data and applying a two-stage classifier with PCA-based aggregation, the system identifies grasp points and plans trajectories for precise manipulation. Experiments show superior textile recognition and sorting performance versus human operators and RGB baselines, with competitive real-time classification speed. The results suggest hyperspectral imaging as a valuable enhancement for robotic perception in challenging environments, while noting limitations in cluttered scenes and opportunities for multi-sensor integration.

Abstract

Hyperspectral imaging is an advanced technique for precisely identifying and analyzing materials or objects. However, its integration with robotic grasping systems has so far been explored due to the deployment complexities and prohibitive costs. Within this paper, we introduce a novel hyperspectral imaging-guided robotic grasping system. The system consists of PRISM (Polyhedral Reflective Imaging Scanning Mechanism) and the SpectralGrasp framework. PRISM is designed to enable high-precision, distortion-free hyperspectral imaging while simplifying system integration and costs. SpectralGrasp generates robotic grasping strategies by effectively leveraging both the spatial and spectral information from hyperspectral images. The proposed system demonstrates substantial improvements in both textile recognition compared to human performance and sorting success rate compared to RGB-based methods. Additionally, a series of comparative experiments further validates the effectiveness of our system. The study highlights the potential benefits of integrating hyperspectral imaging with robotic grasping systems, showcasing enhanced recognition and grasping capabilities in complex and dynamic environments. The project is available at: https://zainzh.github.io/PRISM.

Figures (8)

• Figure 1: Tabletop setup for the hyperspectral textile robotic grasping system. The setup includes a PRISM for hyperspectral image acquisition, halogen lamps, a Bernoulli's Principle suction cup gripper attached to a robotic arm, and a set of wasted textiles (linen, acetate, silk, wool) for sorting.
• Figure 2: System Overview: SpectralGrasp leverages both the spectral and spatial information from hyperspectral images captured by PRISM to generate suction points, enabling robots to complete textile sorting tasks. The process begins with PRISM capturing hyperspectral image frames, which are processed through image reconstruction and distortion correction to produce a corrected hyperspectral image. This corrected image with its mask serves as the input for SpectralGrasp. A pixel-level hyperspectral classifier is applied to generate pixel-level classification results. These results are then aggregated into object-level recognition outcomes using a Principal Component Analysis (PCA) algorithm. The system subsequently identifies the geometric centroids of target objects and generates a series of suction points for each target. A Target Decider module selects the target object, based on the task requirements, to generate the robotic execution trajectory. The trajectory is then sent to the Cartesian Motion Controller for precise sorting and execution.
• Figure 3: CAD renderings of the mechanical design and the working mechanism of PRISM. (A) Fully assembled front view of PRISM, showing the two limited scanning positions. (B) Fully assembled top view. (C) Exploded view of PRISM, illustrating the individual components. (D) The working mechanism of RPISM, highlighting the scanning process. The angle $\theta$ denotes the rotation angle of the servo motor, while the angle $\gamma$ denotes the scanned angle. The circumradius of the rotating prism is $r$, and the position of the hyperspectral sensor, point $A$, is given by $(x_A, \frac{\sqrt{2}r}{2})$.
• Figure 4: Curvature distortion correction. (a) Original image exhibiting distortion with increased scanned angle $\gamma$. (b) Corrected image displaying uniform spatial mapping in both the $x$ and $y$ directions.
• Figure 5: Spatial resolution test results for PRISM at various experimental heights. The resolution was measured using the 1951 USAF Resolution Test Chart (MIL-STD-150), containing reference line patterns with known dimensions. The diagram illustrates the achieved resolution in line pairs per millimeter (lp/mm) at heights ranging from 33 cm to 60 cm. The table below summarizes the spatial resolution performance and minimum resolvable size.