Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Controllable Navigation Instruction Generation with Chain of Thought Prompting

Xianghao Kong, Jinyu Chen, Wenguan Wang, Hang Su, Xiaolin Hu, Yi Yang, Si Liu

TL;DR

C-Instructor is proposed, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation and introduces a Style-Mixed Training policy, harnessing the prior knowledge of LLMs to enable style control for instruction generation based on different prompts within a single model instance.

Abstract

Instruction generation is a vital and multidisciplinary research area with broad applications. Existing instruction generation models are limited to generating instructions in a single style from a particular dataset, and the style and content of generated instructions cannot be controlled. Moreover, most existing instruction generation methods also disregard the spatial modeling of the navigation environment. Leveraging the capabilities of Large Language Models (LLMs), we propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation. Firstly, we propose a Chain of Thought with Landmarks (CoTL) mechanism, which guides the LLM to identify key landmarks and then generate complete instructions. CoTL renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects. Furthermore, we present a Spatial Topology Modeling Task to facilitate the understanding of the spatial structure of the environment. Finally, we introduce a Style-Mixed Training policy, harnessing the prior knowledge of LLMs to enable style control for instruction generation based on different prompts within a single model instance. Extensive experiments demonstrate that instructions generated by C-Instructor outperform those generated by previous methods in text metrics, navigation guidance evaluation, and user studies.

Controllable Navigation Instruction Generation with Chain of Thought Prompting

TL;DR

C-Instructor is proposed, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation and introduces a Style-Mixed Training policy, harnessing the prior knowledge of LLMs to enable style control for instruction generation based on different prompts within a single model instance.

Abstract

Instruction generation is a vital and multidisciplinary research area with broad applications. Existing instruction generation models are limited to generating instructions in a single style from a particular dataset, and the style and content of generated instructions cannot be controlled. Moreover, most existing instruction generation methods also disregard the spatial modeling of the navigation environment. Leveraging the capabilities of Large Language Models (LLMs), we propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation. Firstly, we propose a Chain of Thought with Landmarks (CoTL) mechanism, which guides the LLM to identify key landmarks and then generate complete instructions. CoTL renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects. Furthermore, we present a Spatial Topology Modeling Task to facilitate the understanding of the spatial structure of the environment. Finally, we introduce a Style-Mixed Training policy, harnessing the prior knowledge of LLMs to enable style control for instruction generation based on different prompts within a single model instance. Extensive experiments demonstrate that instructions generated by C-Instructor outperform those generated by previous methods in text metrics, navigation guidance evaluation, and user studies.
Paper Structure (23 sections, 18 equations, 8 figures, 8 tables)

This paper contains 23 sections, 18 equations, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Formulation
  5. Overall Framework
  6. Spatial Topology Modeling Task (STMT)
  7. Chain of Thought with Landmarks (CoTL)
  8. Style-Mixed Training (SMT)
  9. Experiments
  10. Datasets and Evaluation Metrics
  11. Implementation Details
  12. Comparison to State-of-the-Art Methods
  13. Diagnostic Experiment
  14. Instruction Quality Analysis
  15. Qualitative Results
  16. ...and 8 more sections

Figures (8)

  • Figure 1: C-Instructor possesses the ability to control the linguistic style of generated instructions, and the ability to manipulate landmarks within the instructions (§\ref{['sec:intro']}).
  • Figure 2: Overall framework of C-Instructor (§\ref{['sec:framework']}) and details of STMT (§\ref{['sec:stmt']}).
  • Figure 3: Details of Landmark Selection (left) and CoT Inference (right) in CoTL (§\ref{['sec:cotl']}). In Spatial Selection, candidate views are partitioned in blue boxes, and only objects that are distinct in action view are selected as landmarks (marked with a green tick ✓). In Temporal Selection, the action that leads to a new scene is treated as a significant viewpoint (marked in red box).
  • Figure 4: Visualizations of navigation trajectory and instruction generation results on R2R anderson2018vision and REVERIE qi2020reverie (§\ref{['sec:qual']}).
  • Figure 5: Visualizations of path and generated instruction on UrbanWalk huang2022assister (§\ref{['sec:qual']}).
  • ...and 3 more figures