Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning With LLMs Interaction For Distributed Diffusion Model Services

Hongyang Du, Ruichen Zhang, Dusit Niyato, Jiawen Kang, Zehui Xiong, Shuguang Cui, Xuemin Shen, Dong In Kim

TL;DR

This work tackles energy-efficient, user-centric QoE optimization for distributed diffusion-model-based AIGC. It introduces a distributed GDM framework where semantically similar prompts share denoising steps to save energy, and couples this with Reinforcement Learning With LLM Interaction (RLLI) that uses LLM-empowered Generative Agents to provide real-time subjective QoE rewards. The authors develop a GDM-based DDPG variant (G-DDPG-LI) to allocate communication and computing resources while accounting for user personalities and wireless dynamics, achieving up to a 15% QoE gain over conventional DDPG methods. The study demonstrates the viability of edge-enabled, personalized AIGC with efficient resource management in future networks, and points to promising avenues like caching and computation reuse to further enhance performance.

Abstract

Distributed Artificial Intelligence-Generated Content (AIGC) has attracted significant attention, but two key challenges remain: maximizing subjective Quality of Experience (QoE) and improving energy efficiency, which are particularly pronounced in widely adopted Generative Diffusion Model (GDM)-based image generation services. In this paper, we propose a novel user-centric Interactive AI (IAI) approach for service management, with a distributed GDM-based AIGC framework that emphasizes efficient and cooperative deployment. The proposed method restructures the GDM inference process by allowing users with semantically similar prompts to share parts of the denoising chain. Furthermore, to maximize the users' subjective QoE, we propose an IAI approach, i.e., Reinforcement Learning With Large Language Models Interaction (RLLI), which utilizes Large Language Model (LLM)-empowered generative agents to replicate user interaction, providing real-time and subjective QoE feedback aligned with diverse user personalities. Lastly, we present the GDM-based Deep Deterministic Policy Gradient (GDDPG) algorithm, adapted to the proposed RLLI framework, to allocate communication and computing resources effectively while accounting for subjective user traits and dynamic wireless conditions. Simulation results demonstrate that G-DDPG improves total QoE by 15% compared with the standard DDPG algorithm.

Reinforcement Learning With LLMs Interaction For Distributed Diffusion Model Services

TL;DR

This work tackles energy-efficient, user-centric QoE optimization for distributed diffusion-model-based AIGC. It introduces a distributed GDM framework where semantically similar prompts share denoising steps to save energy, and couples this with Reinforcement Learning With LLM Interaction (RLLI) that uses LLM-empowered Generative Agents to provide real-time subjective QoE rewards. The authors develop a GDM-based DDPG variant (G-DDPG-LI) to allocate communication and computing resources while accounting for user personalities and wireless dynamics, achieving up to a 15% QoE gain over conventional DDPG methods. The study demonstrates the viability of edge-enabled, personalized AIGC with efficient resource management in future networks, and points to promising avenues like caching and computation reuse to further enhance performance.

Abstract

Distributed Artificial Intelligence-Generated Content (AIGC) has attracted significant attention, but two key challenges remain: maximizing subjective Quality of Experience (QoE) and improving energy efficiency, which are particularly pronounced in widely adopted Generative Diffusion Model (GDM)-based image generation services. In this paper, we propose a novel user-centric Interactive AI (IAI) approach for service management, with a distributed GDM-based AIGC framework that emphasizes efficient and cooperative deployment. The proposed method restructures the GDM inference process by allowing users with semantically similar prompts to share parts of the denoising chain. Furthermore, to maximize the users' subjective QoE, we propose an IAI approach, i.e., Reinforcement Learning With Large Language Models Interaction (RLLI), which utilizes Large Language Model (LLM)-empowered generative agents to replicate user interaction, providing real-time and subjective QoE feedback aligned with diverse user personalities. Lastly, we present the GDM-based Deep Deterministic Policy Gradient (GDDPG) algorithm, adapted to the proposed RLLI framework, to allocate communication and computing resources effectively while accounting for subjective user traits and dynamic wireless conditions. Simulation results demonstrate that G-DDPG improves total QoE by 15% compared with the standard DDPG algorithm.
Paper Structure (27 sections, 21 equations, 14 figures, 2 tables, 3 algorithms)

This paper contains 27 sections, 21 equations, 14 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Diffusion Models
  4. Aesthetics Analysis and Big Five Personality Traits
  5. LLM-empowered Generative Agents as Evaluators
  6. System Model and Problem Formulation
  7. Distributed Generative Diffusion Model-based AIGC
  8. Latent Diffusion Models
  9. Conditioning Mechanisms
  10. Distributed Diffusion Model Inference
  11. Deployment Method
  12. User-centric QoE Maximization Problem
  13. Reinforcement Learning With LLMs Feedback
  14. Aesthetic-aware QoE Modeling
  15. Big Five Personality Traits
  16. ...and 12 more sections

Figures (14)

  • Figure 1: The basic framework of IAI and four images generated with the prompt "A man sits in the street". Part A is a man engrossed in a book against vibrant street art appeals to users with high openness. Part B is a formally dressed man on a clean street, resonating with users high in conscientiousness. Part C is a sociable street cafe scene, suitable for users with high extraversion. Part D is a man feeding pigeons in a peaceful setting with playing children, fitting for users with high agreeableness.
  • Figure 2: The working principle of the GDM and motivations behind distributed denoising inference process. Part A depicts the cooperative inference process across devices where, starting with Gaussian noise on Device 2, it denoises using Prompt 2 before Devices 1 and $3$ continue in succession towards their respective prompts. Part B shows the fundamentals of distributed GDM inference, illustrating how denoising path directions alter with changing prompts, and emphasizing the high semantic similarity of Prompt 1 to Prompt 2, contrasting it with the distance between Prompt 2 and Prompt 3. Consequently, Path 2 aligns with Prompt 1's requirements, unlike Path 4 with Prompt 3. Part C showcases the final generation outcomes for all five paths.
  • Figure 3: The working steps of GDM-based text-to-image generation service, and the principle to achieve multi-device distributed denoising process.
  • Figure 4: Deployment method for distributed GDM-based text-to-image AIGC services: Part A illustrates the shared inference mechanism where users with semantically similar prompts collaborate with the server. Part B highlights the diverse text-to-image requirements of each of the $K$ concurrent users. Following initial server processing, Part D delves into the distributed denoising process by each individual user. Part C presents the final generated images corresponding to user-specific prompts.
  • Figure 5: Five types of personality traits in the Big-Five personality model: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. We show real user personality scores from the PsychoFlickr database, along with examples of the images that they prefer.
  • ...and 9 more figures