Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis

Hanyu Jiang, Jian Xue, Xing Lan, Guohong Hu, Ke Lu

TL;DR

MVLLaVA addresses the challenge of flexible and unified novel view synthesis by integrating multiple multi-view diffusion models with a vision-language backbone (LLaVA). It introduces task-specific instruction templates and an efficient LoRA-based fine-tuning regime to enable a single agent to interpret natural-language instructions and select the appropriate downstream diffusion model. Empirical results show robust task recognition, accurate azimuth control, and competitive performance against state-of-the-art large multimodal models, while maintaining a lightweight 7B base. The framework offers a scalable, user-friendly approach for unified multi-view generation across diverse inputs and instructions, with practical implications for interactive view synthesis and content creation.

Abstract

This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges.

MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis

TL;DR

MVLLaVA addresses the challenge of flexible and unified novel view synthesis by integrating multiple multi-view diffusion models with a vision-language backbone (LLaVA). It introduces task-specific instruction templates and an efficient LoRA-based fine-tuning regime to enable a single agent to interpret natural-language instructions and select the appropriate downstream diffusion model. Empirical results show robust task recognition, accurate azimuth control, and competitive performance against state-of-the-art large multimodal models, while maintaining a lightweight 7B base. The framework offers a scalable, user-friendly approach for unified multi-view generation across diverse inputs and instructions, with practical implications for interactive view synthesis and content creation.

Abstract

This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges.
Paper Structure (19 sections, 9 figures, 4 tables)

This paper contains 19 sections, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-view Diffusion Models
  4. Large Multimodal Models with Vision Instruction Tuning
  5. MVLLaVA
  6. Overview
  7. View-based Instructions
  8. Image-based tasks
  9. Caption-based tasks
  10. Related-view-based task
  11. Answer format
  12. Efficient Fine-tuning
  13. Experiments
  14. Implementation Details
  15. Ablation Study
  16. ...and 4 more sections

Figures (9)

  • Figure 1: Application cases of MVLLaVA in novel view synthesis: MVLLaVA generates novel view images around an object or from specified viewpoints, using a reference image or caption. It can also rotate the camera to produce the desired image. MVLLaVA unifies multi-view generation capabilities to synthesize novel views, with user-friendly and intuitive instructions.
  • Figure 2: The overall architecture of MVLLaVA. It consists of three main components: a large multimodal model LLaVA, a post-processing module, and multi-view diffusion models.
  • Figure 3: Training of LLaVA. We freeze the language model and the visual encoder, adding the LoRA module into both the visual encoder and the LLM.
  • Figure 4: The Qualitative results of MVLLaVA across five different tasks, where "I" denotes "Img-2-3d" and "T" denotes "Text-2-3d".
  • Figure 5: The Qualitative results of MVLLaVA on the Img-2-3d-around task. ImageDreamwang2023imagedream is used for multi-view diffusion.
  • ...and 4 more figures