A Roadmap Towards Automated and Regulated Robotic Systems

TL;DR

It is argued that the unregulated generative processes from AI is fitted for low level end tasks, but intervention in the form of manual or automated regulation should happen post-workflow-generation and pre-robotic-execution.

Abstract

The rapid development of generative technology opens up possibility for higher level of automation, and artificial intelligence (AI) embodiment in robotic systems is imminent. However, due to the blackbox nature of the generative technology, the generation of the knowledge and workflow scheme is uncontrolled, especially in a dynamic environment and a complex scene. This poses challenges to regulations in safety-demanding applications such as medical scenes. We argue that the unregulated generative processes from AI is fitted for low level end tasks, but intervention in the form of manual or automated regulation should happen post-workflow-generation and pre-robotic-execution. To address this, we propose a roadmap that can lead to fully automated and regulated robotic systems. In this paradigm, the high level policies are generated as structured graph data, enabling regulatory oversight and reusability, while the code base for lower level tasks is generated by generative models. Our approach aims the transitioning from expert knowledge to regulated action, akin to the iterative processes of study, practice, scrutiny, and execution in human tasks. We identify the generative and deterministic processes in a design cycle, where generative processes serve as a text-based world simulator and the deterministic processes generate the executable system. We propose State Machine Seralization Language (SMSL) to be the conversion point between text simulator and executable workflow control. From there, we analyze the modules involved based on the current literature, and discuss human in the loop. As a roadmap, this work identifies the current possible implementation and future work. This work does not provide an implemented system but envisions to inspire the researchers working on the direction in the roadmap. We implement the SMSL and D-SFO paradigm that serve as the starting point of the roadmap.

Figures (9)

• Figure 1: Automatically solving Hanoi Tower using da Vinci Master Tool Manipulator (MTM) in dVRK, with the state transitions controlled by SMSL.
• Figure 2: The design cycle and the architecture of a process-controlled medical robotic system liu2023toward. The panel Design Cycle identifies the generative processes and the deterministic processes in the design cycle. Consultation with clinicians, FSM design, SB identification, and hFSM design are generative processes because they are dependent on expert experience and training. Once the hFSM design is completed, the modeling of the automation (states and their transitions) are determined. Thus, the processes followed are deterministic processes. hFSM classes implementation and D-SFO (DispatcherState/Flag/Operation paradigm) implementation are the processes that generate the codebase running in the designed system. These processes can be automated because the coded components are standardized. System testing and hardware integration may be semi-deterministic because of the variations of deployed environment. However, if the world modeling is optimized, these two processes should also be able to be optimized. The panel Designed System shows the architecture of the produced system.
• Figure 3: The architecture of the proposed paradigm, using medical robotics as an example. The modules with a green check icon are the ones that can be implemented using systems proposed in the current literature. The module with a red question mark icon is future work. The architecture approaches the intelligent robotic system in three divisions. The upstream division handles the generation of the Expert Text (ET), the midstream division involves the automatic translation of the ET to data structures in programming languages, and the downstream produces machine-executable codes for the robotic procedure, that are tailored to hardware and testing differences.
• Figure 4: Example FSMs for Hanoi Tower game and for medical image registration process. The Hanoi Tower panel shows two cases of the game: Two disks in three possible positions, and three disks in three possible positions. Fig. \ref{['fig:davincihanoi']} shows a simulation using the da Vinci Master Tool Manipulator (MTM) in da Vinci Research Kit (dVRK) to play Hanoi Tower automatically. The Medical Image Registration panel is from liu2023toward, copyright by Liu et al. and adapted with permission. The corresponding SMSLs for both Hanoi Tower and medical image registration are listed in Appendix \ref{['sec:hanoismsl']} and \ref{['sec:registrationsmsl']}. Both examples here are not hierarchical. liu2023toward shows the hFSM for medical image registration. For an additional simple hFSM example, Appendix \ref{['sec:hfsmsmsl']} lists the graph and the corresponding SMSL.
• Figure 5: Modeling of the scene using lambda algebra.