Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pre-Filtering Code Suggestions using Developer Behavioral Telemetry to Optimize LLM-Assisted Programming

Mohammad Nour Al Awad, Sergey Ivanov, Olga Tikhonova

TL;DR

This work tackles wasted computation and interruptions in LLM-assisted programming by introducing a lightweight pre-filter that gates LLM invocation using real-time developer telemetry. The behavioral-only gate operates before any prompt or code content is analyzed, preserving privacy while remaining language-agnostic. Trained on 2,318 suggestion events and deployed in a production VS Code plugin, the approach increased suggestion acceptance from $18.4\%$ to $34.2\%$ and suppressed about $35\%$ of low-value LLM calls, without modifying the LLM itself. These results demonstrate that timing-aware, privacy-preserving adaptation based on behavioral signals can substantially improve both user experience and system efficiency in AI-assisted programming, and point to future work on personalization and richer developer-state modeling.

Abstract

Large Language Models (LLMs) are increasingly integrated into code editors to provide AI-powered code suggestions. Yet many of these suggestions are ignored, resulting in wasted computation, increased latency, and unnecessary interruptions. We introduce a lightweight pre-filtering model that predicts the likelihood of suggestion acceptance before invoking the LLM, using only real-time developer telemetry such as typing speed, file navigation, and editing activity. Deployed in a production-grade Visual Studio Code plugin over four months of naturalistic use, our approach nearly doubled acceptance rates (18.4% -> 34.2%) while suppressing 35% of low-value LLM calls. These findings demonstrate that behavioral signals alone can meaningfully improve both user experience and system efficiency in LLM-assisted programming, highlighting the value of timing-aware, privacy-preserving adaptation mechanisms. The filter operates solely on pre-invocation editor telemetry and never inspects code or prompts.

Pre-Filtering Code Suggestions using Developer Behavioral Telemetry to Optimize LLM-Assisted Programming

TL;DR

This work tackles wasted computation and interruptions in LLM-assisted programming by introducing a lightweight pre-filter that gates LLM invocation using real-time developer telemetry. The behavioral-only gate operates before any prompt or code content is analyzed, preserving privacy while remaining language-agnostic. Trained on 2,318 suggestion events and deployed in a production VS Code plugin, the approach increased suggestion acceptance from to and suppressed about of low-value LLM calls, without modifying the LLM itself. These results demonstrate that timing-aware, privacy-preserving adaptation based on behavioral signals can substantially improve both user experience and system efficiency in AI-assisted programming, and point to future work on personalization and richer developer-state modeling.

Abstract

Large Language Models (LLMs) are increasingly integrated into code editors to provide AI-powered code suggestions. Yet many of these suggestions are ignored, resulting in wasted computation, increased latency, and unnecessary interruptions. We introduce a lightweight pre-filtering model that predicts the likelihood of suggestion acceptance before invoking the LLM, using only real-time developer telemetry such as typing speed, file navigation, and editing activity. Deployed in a production-grade Visual Studio Code plugin over four months of naturalistic use, our approach nearly doubled acceptance rates (18.4% -> 34.2%) while suppressing 35% of low-value LLM calls. These findings demonstrate that behavioral signals alone can meaningfully improve both user experience and system efficiency in LLM-assisted programming, highlighting the value of timing-aware, privacy-preserving adaptation mechanisms. The filter operates solely on pre-invocation editor telemetry and never inspects code or prompts.

Paper Structure

This paper contains 39 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI-Powered Code Suggestions and Developer Acceptance
  4. Telemetry-Driven Observability in IDEs
  5. Modeling Developer Behavior
  6. Interruptibility and Context-Aware Assistance
  7. Positioning and Novelty
  8. Behavioral Metrics and Modeling
  9. System Architecture
  10. Data Collection Procedure
  11. Per-Suggestion Logging.
  12. Continuous Behavioral Metrics.
  13. Estimating Task Complexity.
  14. Behavioral Categories and Feature Design
  15. Model Training
  16. ...and 24 more sections

Figures (2)

  • Figure 1: SHAP value analysis for the CatBoost acceptance classifier.
  • Figure 2: Suggestion filtering decisions over the course of a real developer session. The model dynamically alternates between allowing and suppressing requests in response to real-time behavior. Extended filtering often coincides with focused activity, while suggestions tend to be allowed during pauses or fragmented interaction.