Domain-Specific Fine-Tuning of Large Language Models for Interactive Robot Programming

TL;DR

The paper addresses the challenge of making industrial robot programming accessible via natural language by studying domain-specific fine-tuning of foundation models under data and compute constraints. It compares three training paradigms (instruction-following, streaming adaptation, and a two-stage streaming-plus-instruction approach) using Alpaca and LLaMA families with QLoRA and 4-bit NormalFloat quantization, plus prefix-tuning, on domain datasets derived from ArtiMinds RPS. Across semantic similarity (BERTScore) and expert surveys, domain-specific instruction-following yields the strongest transfer, but models still hallucinate and diverge from domain requirements, while prefix-tuning shows limited gains. The findings suggest that while domain adaptation helps, prompting strategies and larger-scale evaluations are essential for reliable, real-world NL-based robot programming in resource-constrained industrial settings.

Abstract

Industrial robots are applied in a widening range of industries, but robot programming mostly remains a task limited to programming experts. We propose a natural language-based assistant for programming of advanced, industrial robotic applications and investigate strategies for domain-specific fine-tuning of foundation models with limited data and compute.