OPTIAGENT: A Physics-Driven Agentic Framework for Automated Optical Design

TL;DR

This work bridges the expertise gap by enabling users without formal optical training to successfully develop functional lens systems by injecting domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion.

Abstract

Optical design is the process of configuring optical elements to precisely manipulate light for high-fidelity imaging. It is inherently a highly non-convex optimization problem that relies heavily on human heuristic expertise and domain-specific knowledge. While Large Language Models (LLMs) possess extensive optical knowledge, their capabilities in leveraging the knowledge in designing lens system remain significantly constrained. This work represents the first attempt to employ LLMs in the field of optical design. We bridge the expertise gap by enabling users without formal optical training to successfully develop functional lens systems. Concretely, we curate a comprehensive dataset, named OptiDesignQA, which encompasses both classical lens systems sourced from standard optical textbooks and novel configurations generated by automated design algorithms for training and evaluation. Furthermore, we inject domain-specific optical expertise into the LLM through a hybrid objective of full-system synthesis and lens completion. To align the model with optical principles, we employ Group Relative Policy Optimization Done Right (DrGRPO) guided by Optical Lexicographic Reward for physics-driven policy alignment. This reward system incorporates structural format rewards, physical feasibility rewards, light-manipulation accuracy, and LLM-based heuristics. Finally, our model integrates with specialized optical optimization routines for end-to-end fine-tuning and precision refinement. We benchmark our proposed method against both traditional optimization-based automated design algorithms and LLM counterparts, and experimental results show the superiority of our method.

Figures (5)

• Figure 1: Despite possessing foundational optical knowledge, general-purpose LLMs (e.g., ChatGPT) fail to generate physically realizable systems, regardless of the prompting strategy. In contrast, OptiAgent bridges this gap by producing designs that strictly satisfy both practical and physical constraints.
• Figure 2: Overview of the OptiAgent framework. Given natural language instruction, the system integrates both optical prescription completion task and whole optical system design task, a physics-driven policy with hierarchical rewards, and a function call of Zemax in a closed loop to synthesize optical prescriptions $\mathcal{L}$ from specifications $\mathcal{P}$. At inference, the generated initial structure $\mathcal{L}_0$ undergoes Zemax local optimization for final refinement.
• Figure 3: Statistics of EFFL, FoV, and F-number in our OptiDesignQA dataset. Our dataset covers a broad spectrum of specifications for commonly used optical lenses. Distinct peaks in F-number arises from the prevalence of standard engineering aperture settings in practical optical designs.
• Figure 4: Layouts of representative designs from all evaluated methods. For each optical system, the RMS spot diagrams for three distinct FoVs are displayed. The first row illustrates the initial layouts produced by each method before fine-grained optimization in Zemax, while the second row shows the corresponding results after optimization. Ray colors blue, green, and red represent wavelengths of 0.485 $\mu$m, 0.588 $\mu$m, and 0.656 $\mu$m, respectively. Even when compared against significantly larger architectures like Qwen3-235B, our method utilizing Qwen3-4B as a foundation model exhibits better performance.
• Figure 5: Illustration of an imaging optical system.