ChunkWise LoRA: Adaptive Sequence Partitioning for Memory-Efficient Low-Rank Adaptation and Accelerated LLM Inference

TL;DR

ChunkWise LoRA tackles the inefficiency of uniform LoRA by partitioning input sequences into chunks with variable complexity and assigning per-chunk LoRA rank and scaling. The framework combines a token complexity estimator, adaptive chunking, a rank ladder-based selector, boundary-safe composition, and KV-cache policies to deliver up to $34\%$ lower latency and $38\%$ memory reduction while maintaining or improving task performance. It remains fully compatible with standard transformer cores, mixed precision, and existing inference stacks, enabling practical deployment without retraining. By coupling adaptive capacity allocation with memory-aware cache strategies, the approach offers a scalable solution for efficient real-world LLM inference across edge and cloud settings.

Abstract

Recent advances in low-rank adaptation (LoRA) have enabled efficient fine-tuning of large language models (LLMs) with minimal additional parameters. However, existing LoRA methods apply static rank configurations uniformly across all input tokens, ignoring variation in token complexity and computational requirements. In this work, we propose ChunkWise LoRA, a dynamic and adaptive approach that partitions sequences into variable-length chunks based on token complexity and assigns each chunk a tailored low-rank configuration. Our system introduces a runtime scheduler that estimates token difficulty, performs adaptive chunking, and selects per-chunk LoRA rank and scaling using a rank-ladder mechanism. To preserve output consistency, we further introduce a boundary-safe composition module and integrate policy-driven KV-cache strategies. Experiments on benchmark datasets such as Wikitext-103 and SQuAD demonstrate that ChunkWise LoRA achieves up to 34\% lower latency and 38% memory reduction compared to baseline LoRA, while maintaining or improving task performance metrics like BLEU, EM, and perplexity. The proposed framework remains fully compatible with existing transformer architectures and inference frameworks, providing a practical solution for real-world deployment of parameter-efficient LLMs.

Figures (3)

• Figure 1: System architecture of ChunkWise LoRA
• Figure 2: Latency vs. Perplexity across fine-tuning strategies.
• Figure 3: Memory usage vs. BLEU score comparison.