Optimizing LLM Queries in Relational Data Analytics Workloads

TL;DR

The document outlines MLSys 2024 submission and formatting instructions, detailing an entirely electronic, double-blind review process with strict page, font, and layout requirements. It prescribes a maximum of 10 pages for the main content on US Letter, two-column pages using 10-point Times with embedded Type-1 fonts, and provides detailed guidance for anonymized submissions and camera-ready copies. The guidelines cover abstract structure, sectioning, figures, tables, algorithms, citations (APA style via natbib), and the optional inclusion of software and data links or supplements, all aimed at ensuring consistency and reproducibility. Together, these rules establish a rigorous template to facilitate fair evaluation and uniform presentation across MLSys submissions.

Abstract

Batch data analytics is a growing application for Large Language Models (LLMs). LLMs enable users to perform a wide range of natural language tasks, such as classification, entity extraction, and translation, over large datasets. However, LLM inference is highly costly and slow: for example, an NVIDIA L4 GPU running Llama3-8B can only process 6 KB of text per second, taking about a day to handle 15 GB of data; processing a similar amount of data costs around $10K on OpenAI's GPT-4o. In this paper, we propose novel techniques that can significantly reduce the cost of LLM calls for relational data analytics workloads. Our key contribution is developing efficient algorithms for reordering the rows and the fields within each row of an input table to maximize key-value (KV) cache reuse when performing LLM serving. As such, our approach can be easily applied to existing analytics systems and serving platforms. Our evaluation shows that our solution can yield up to 3.4x improvement in job completion time on a benchmark of diverse LLM-based queries using Llama 3 models. Our solution also achieves a 32% cost savings under OpenAI and Anthropic pricing models.