Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks

TL;DR

This work addresses efficient, task-driven communication for large multimodal vehicle AI by placing the heavy LLM workload on cloud servers and keeping image encoding lightweight on the vehicle. It introduces a Semantic Matching–based image slicing strategy (SM) to reduce visual tokens, and a fusion attention mechanism (FA-SemCom) that combines objective saliency with user-specific attention to allocate transmission power across image patches. Empirical results on a traffic VQA dataset show substantial accuracy gains in low-SNR regimes and faster responsiveness, underscoring the value of semantic prioritization and attention-guided resource allocation for edge-LLM systems in dynamic vehicular networks. The approach is extensible to other domains, with potential benefits in bandwidth efficiency, privacy, and real-time reliability for multimodal intelligent systems.

Abstract

Task-oriented semantic communication has emerged as a fundamental approach for enhancing performance in various communication scenarios. While recent advances in Generative Artificial Intelligence (GenAI), such as Large Language Models (LLMs), have been applied to semantic communication designs, the potential of Large Multimodal Models (LMMs) remains largely unexplored. In this paper, we investigate an LMM-based vehicle AI assistant using a Large Language and Vision Assistant (LLaVA) and propose a task-oriented semantic communication framework to facilitate efficient interaction between users and cloud servers. To reduce computational demands and shorten response time, we optimize LLaVA's image slicing to selectively focus on areas of utmost interest to users. Additionally, we assess the importance of image patches by combining objective and subjective user attention, adjusting energy usage for transmitting semantic information. This strategy optimizes resource utilization, ensuring precise transmission of critical information. We construct a Visual Question Answering (VQA) dataset for traffic scenarios to evaluate effectiveness. Experimental results show that our semantic communication framework significantly increases accuracy in answering questions under the same channel conditions, performing particularly well in environments with poor Signal-to-Noise Ratios (SNR). Accuracy can be improved by 13.4% at an SNR of 12dB and 33.1% at 10dB, respectively.

Figures (9)

• Figure 1: The proposed task-oriented semantic communication system framework. Section \ref{['S41']} introduces the architecture of LLaVA, Section \ref{['S42']} presents the optimization method for image encoding, and Section \ref{['S5']} details the proposed fusion attention-based power allocation method.
• Figure 2: A simple processing flow of the system framework
• Figure 3: Three different image encoding mechanisms (i.e., LLaVA-1.5, LLaVA-1.6, and the proposed LLaVA-SM). The visualization displays the clarity achieved by each method in critical areas of the image.
• Figure 4: The proposed fusion attention-based resource allocation method. By evaluating the attention weights of each image patch, resources are concentrated on the critical parts of the image.
• Figure 5: Comparison of average response times for LLaVA-1.6 and LLaVA-SM.