LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning

TL;DR

LEGO advances egocentric action frame generation by enriching action prompts through visual instruction tuning of a Visual Large Language Model and by incorporating VLLM-derived image and text embeddings as conditioning in a latent diffusion model. This approach narrows the domain gap between exocentric diffusion pretraining and egocentric action synthesis, enabling faithful, context-preserving frames conditioned on user prompts. Thorough experiments on Ego4D and Epic-Kitchens, along with extensive ablations, demonstrate superior performance over baselines across image-to-image and image-to-text metrics and via user studies, while analyses highlight the critical role of prompt enrichment and VLLM finetuning. The work offers a scalable path toward actionable egocentric guidance and skill transfer through visually guided, context-aware frame generation.

Abstract

Generating instructional images of human daily actions from an egocentric viewpoint serves as a key step towards efficient skill transfer. In this paper, we introduce a novel problem -- egocentric action frame generation. The goal is to synthesize an image depicting an action in the user's context (i.e., action frame) by conditioning on a user prompt and an input egocentric image. Notably, existing egocentric action datasets lack the detailed annotations that describe the execution of actions. Additionally, existing diffusion-based image manipulation models are sub-optimal in controlling the state transition of an action in egocentric image pixel space because of the domain gap. To this end, we propose to Learn EGOcentric (LEGO) action frame generation via visual instruction tuning. First, we introduce a prompt enhancement scheme to generate enriched action descriptions from a visual large language model (VLLM) by visual instruction tuning. Then we propose a novel method to leverage image and text embeddings from the VLLM as additional conditioning to improve the performance of a diffusion model. We validate our model on two egocentric datasets -- Ego4D and Epic-Kitchens. Our experiments show substantial improvement over prior image manipulation models in both quantitative and qualitative evaluation. We also conduct detailed ablation studies and analysis to provide insights in our method. More details of the dataset and code are available on the website (https://bolinlai.github.io/Lego_EgoActGen/).

Figures (20)

• Figure 1: When a user performing a complex task asks a large language model (LLM) for instructions (a) on how to complete the steps, she receives a generic answer and has to translate the guidance into her specific situation. If she is wearing a camera, then the prompt can be augmented with an egocentric view of the scene and passed to a Visual LLM (b), and the description is now contextualized to her situation. But she still faces the challenge of parsing a written description. When she uses our novel LEGO model (c), however, the combined image and prompt are used to automatically generate an image that provides visual guidance exactly in her situation from the egocentric viewpoint. Now she can complete her task seamlessly!
• Figure 2: Examples for data curation using GPT-3.5. We provide detailed action descriptions of several example images as well as their action labels and bounding boxes for in-context learning. In addition, we input the action label and bounding boxes of another action as a query. GPT-3.5 is able to generate descriptions with enriched information (highlighted by underlines) in the response.
• Figure 3: Overview of our proposed LEGO model. We first finetune a visual large language model (VLLM) to generate the enriched action description with visual instruction tuning. We then project image and text embeddings from the finetuned VLLM to the feature space of the latent diffusion model (LDM). Finally, we train the LDM to synthesize the egocentric action frame conditioning on the input frame, enriched action description, as well as the VLLM image and text embeddings.
• Figure 4: Image-to-text metrics of our model and baselines. The best results are highlighted with boldface. The orange row refers to our LEGO model performance.
• Figure 5: User study of our model and baselines. Win rate is the percentage of each model being picked as the best.