Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoning

Kai Zhang, Xingyu Chen, Xiaofeng Zhang

TL;DR

AdaToken-3D tackles inefficiencies in 3D large multimodal models by dynamically pruning spatial tokens based on quantified information contributions across layers and modalities. It introduces intra- and inter-modal contribution metrics and a derivative-constrained optimization with an exponential layer-wise token-retention schedule, enabling layer-aware pruning that preserves reasoning while reducing computation. Empirical results on LLaVA-3D-7B show up to ~63% FLOPs reduction and substantial latency savings with maintained task accuracy, along with insights that a large majority of spatial tokens contribute minimally to final predictions. The work highlights distinct spatial information flow in 3D LMMs compared to 2D counterparts and lays a foundation for efficient, scalable 3D multimodal reasoning toward embodied AI.

Abstract

Large Multimodal Models (LMMs) have become a pivotal research focus in deep learning, demonstrating remarkable capabilities in 3D scene understanding. However, current 3D LMMs employing thousands of spatial tokens for multimodal reasoning suffer from critical inefficiencies: excessive computational overhead and redundant information flows. Unlike 2D VLMs processing single images, 3D LMMs exhibit inherent architectural redundancy due to the heterogeneous mechanisms between spatial tokens and visual tokens. To address this challenge, we propose AdaToken-3D, an adaptive spatial token optimization framework that dynamically prunes redundant tokens through spatial contribution analysis. Our method automatically tailors pruning strategies to different 3D LMM architectures by quantifying token-level information flows via attention pattern mining. Extensive experiments on LLaVA-3D (a 7B parameter 3D-LMM) demonstrate that AdaToken-3D achieves 21\% faster inference speed and 63\% FLOPs reduction while maintaining original task accuracy. Beyond efficiency gains, this work systematically investigates redundancy patterns in multimodal spatial information flows through quantitative token interaction analysis. Our findings reveal that over 60\% of spatial tokens contribute minimally ($<$5\%) to the final predictions, establishing theoretical foundations for efficient 3D multimodal learning.

AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoning

TL;DR

AdaToken-3D tackles inefficiencies in 3D large multimodal models by dynamically pruning spatial tokens based on quantified information contributions across layers and modalities. It introduces intra- and inter-modal contribution metrics and a derivative-constrained optimization with an exponential layer-wise token-retention schedule, enabling layer-aware pruning that preserves reasoning while reducing computation. Empirical results on LLaVA-3D-7B show up to ~63% FLOPs reduction and substantial latency savings with maintained task accuracy, along with insights that a large majority of spatial tokens contribute minimally to final predictions. The work highlights distinct spatial information flow in 3D LMMs compared to 2D counterparts and lays a foundation for efficient, scalable 3D multimodal reasoning toward embodied AI.

Abstract

Large Multimodal Models (LMMs) have become a pivotal research focus in deep learning, demonstrating remarkable capabilities in 3D scene understanding. However, current 3D LMMs employing thousands of spatial tokens for multimodal reasoning suffer from critical inefficiencies: excessive computational overhead and redundant information flows. Unlike 2D VLMs processing single images, 3D LMMs exhibit inherent architectural redundancy due to the heterogeneous mechanisms between spatial tokens and visual tokens. To address this challenge, we propose AdaToken-3D, an adaptive spatial token optimization framework that dynamically prunes redundant tokens through spatial contribution analysis. Our method automatically tailors pruning strategies to different 3D LMM architectures by quantifying token-level information flows via attention pattern mining. Extensive experiments on LLaVA-3D (a 7B parameter 3D-LMM) demonstrate that AdaToken-3D achieves 21\% faster inference speed and 63\% FLOPs reduction while maintaining original task accuracy. Beyond efficiency gains, this work systematically investigates redundancy patterns in multimodal spatial information flows through quantitative token interaction analysis. Our findings reveal that over 60\% of spatial tokens contribute minimally (5\%) to the final predictions, establishing theoretical foundations for efficient 3D multimodal learning.
Paper Structure (14 sections, 9 equations, 3 figures, 4 tables)

This paper contains 14 sections, 9 equations, 3 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. 3D Large-Multimodal-Models
  4. Information Redundancy in LVLMs
  5. Methods
  6. Inefficient Spatial Attention in 3D LMMs
  7. Calculation of Information Contribution
  8. Adaptive Pruning Strategy for Spatial Tokens.
  9. Experiments
  10. Setup
  11. Performance of AdaToken-3D in inference
  12. Assessing the Inference Efficiency of AdaToken-3D
  13. Comparison on 2D benchmarks
  14. CONCLUSIONS

Figures (3)

  • Figure 1: Overview of Spatial tokens in 3D LMMs. Block (A) demonstrates that LLaVA-3D is injected with 3D patches to acquire the 3D scene understanding while maintaining the lightweight architecture as 2D LLMs. LLaVA-3D could achieve state-of-the-art performance across a wide range of 3D benchmarks(3DQA 3DVG). The right Blocks (B) and (C) have pointed out the spatial tokens id index and different types of token processed in LLaVA-3D. Then comparing the information contribution of various tokens in different layers.
  • Figure 2: Observations about Spatial Information Flow in 3D LMMs. (A) is the SQA3D performance of LLaVA-3D-7B with varying ration of pruned spatial tokens at different layers. Spatial-token ranking is based on its Attention Score. (B) is the attention map of system tokens,spatial tokens, prompt tokens and answer tokens. Spatial tokens, just like in 2D LLMs, get low attention allocation in deep layers(3-29). Notably, spatial tokens produce a large proportion of information redundancy in shallow layers, it confirms that the limited attention of the model cannot be spread across multiple input tokens.
  • Figure 3: AdaToken-3D Pruning Strategy. We extract information in various forms using modal characteristics, and layer by layer, calculate the contribution of spatial tokens. This strategy, grounded in a lightweight pruning ratio mechanism, effectively reduces spatial tokens and selects crucial ones relevant to instructions.The pruning results on the right reflect the significance and smoothness of the strategy effect.