Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Idea to Co-Creation: A Planner-Actor-Critic Framework for Agent Augmented 3D Modeling

Jin Gao, Saichandu Juluri

TL;DR

This work proposes a Planner-Actor-Critic framework for agent-augmented 3D modeling that integrates human-in-the-loop supervision to improve geometric and visual quality compared to single-prompt approaches. By coordinating a planning agent, an execution agent, and a critic agent within a Blender-MCP environment and real-time web interface, the system enables iterative refinement and user-guided creativity. Empirical evaluation on three low-poly tasks demonstrates improved fidelity, material assignment, and task completion, while highlighting the benefits of human oversight in steering iterations and preventing stagnation. The findings suggest that structured self-reflection in multi-agent setups, augmented by user input, can produce higher-quality 3D models with efficient workflow integration, albeit with limitations in tool diversity and occasional feedback integration.

Abstract

We present a framework that extends the Actor-Critic architecture to creative 3D modeling through multi-agent self-reflection and human-in-the-loop supervision. While existing approaches rely on single-prompt agents that directly execute modeling commands via tools like Blender MCP, our approach introduces a Planner-Actor-Critic architecture. In this design, the Planner coordinates modeling steps, the Actor executes them, and the Critic provides iterative feedback, while human users act as supervisors and advisors throughout the process. Through systematic comparison between single-prompt modeling and our reflective multi-agent approach, we demonstrate improvements in geometric accuracy, aesthetic quality, and task completion rates across diverse 3D modeling scenarios. Our evaluation reveals that critic-guided reflection, combined with human supervisory input, reduces modeling errors and increases complexity and quality of the result compared to direct single-prompt execution. This work establishes that structured agent self-reflection, when augmented by human oversight and advisory guidance, produces higher-quality 3D models while maintaining efficient workflow integration through real-time Blender synchronization.

From Idea to Co-Creation: A Planner-Actor-Critic Framework for Agent Augmented 3D Modeling

TL;DR

This work proposes a Planner-Actor-Critic framework for agent-augmented 3D modeling that integrates human-in-the-loop supervision to improve geometric and visual quality compared to single-prompt approaches. By coordinating a planning agent, an execution agent, and a critic agent within a Blender-MCP environment and real-time web interface, the system enables iterative refinement and user-guided creativity. Empirical evaluation on three low-poly tasks demonstrates improved fidelity, material assignment, and task completion, while highlighting the benefits of human oversight in steering iterations and preventing stagnation. The findings suggest that structured self-reflection in multi-agent setups, augmented by user input, can produce higher-quality 3D models with efficient workflow integration, albeit with limitations in tool diversity and occasional feedback integration.

Abstract

We present a framework that extends the Actor-Critic architecture to creative 3D modeling through multi-agent self-reflection and human-in-the-loop supervision. While existing approaches rely on single-prompt agents that directly execute modeling commands via tools like Blender MCP, our approach introduces a Planner-Actor-Critic architecture. In this design, the Planner coordinates modeling steps, the Actor executes them, and the Critic provides iterative feedback, while human users act as supervisors and advisors throughout the process. Through systematic comparison between single-prompt modeling and our reflective multi-agent approach, we demonstrate improvements in geometric accuracy, aesthetic quality, and task completion rates across diverse 3D modeling scenarios. Our evaluation reveals that critic-guided reflection, combined with human supervisory input, reduces modeling errors and increases complexity and quality of the result compared to direct single-prompt execution. This work establishes that structured agent self-reflection, when augmented by human oversight and advisory guidance, produces higher-quality 3D models while maintaining efficient workflow integration through real-time Blender synchronization.
Paper Structure (25 sections, 4 figures, 1 table)

This paper contains 25 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. System Overview
  5. Agent Definition
  6. Tooling and Integration
  7. Experiments and Evaluation
  8. Tasks and Setup
  9. Evaluation Metrics
  10. Geometric Quality
  11. Visual Quality
  12. Discussion & Future Works
  13. Efficacy of the Self-Reflective Framework
  14. Emergent Behaviors and Human Oversight
  15. Limitations and Future Improvements
  16. ...and 10 more sections

Figures (4)

  • Figure 1: System Diagram
  • Figure 2: Agent-Human Co-creation Loop
  • Figure 3: (Top) Multi-Agent Co-Modeling Interface. (Bottom, left to right) Examples of: manual bi-directional object editing between Blender and Three.js, wireframe mode, and Blender scene being synced in real time.
  • Figure 4: Geometric Quality Analysis in Autonomous Modeling Tasks