Codetations: Intelligent, Persistent Notes and UIs for Programs and Other Documents

TL;DR

Codetations addresses the problem of missing, duct-taped programming context by introducing document-external, semantically anchored annotations that persist through code evolution. Implemented as a VSCode extension, the system uses real-time edit tracking and LLM-assisted reattachment to keep annotations in sync, while offering interactive annotation types that can execute or test code alongside the host document. A user study with nine participants shows non-intrusive annotations and robust code-tracking are highly valued, along with rapid generation of annotation types via LLM prompts. LLM evaluations indicate that contextual annotations substantially improve bug repair reasoning, though care must be taken to curate relevant context and avoid distraction from irrelevant data. Overall, Codetations demonstrates the feasibility and positive reception of persistent, developer-facing annotations that augment code understanding and AI-assisted development, with clear paths for scaling, language support, and richer annotation types.

Abstract

Software developers maintain extensive mental models of code they produce and its context, often relying on memory to retrieve or reconstruct design decisions, edge cases, and debugging experiences. These missing links and data obstruct both developers and, more recently, large language models (LLMs) working with unfamiliar code. We present Codetations, a system that helps developers contextualize documents with rich notes and tools. Unlike previous approaches, notes in Codetations stay outside the document to prevent code clutter, attaching to spans in the document using a hybrid edit-tracking/LLM-based method. Their content is dynamic, interactive, and synchronized with code changes. A worked example shows that relevant notes with interactively-collected data improve LLM performance during code repair. In our user evaluation, developers praised these properties and saw significant potential in annotation types that we generated with an LLM in just a few minutes.

Figures (4)

• Figure 1: (right) Codetations connects the external Odyssey floating-point workbench, hosted on the web or in an editor, to (left) source code. An Analyze Floating-point Expression annotation uses an LLM to translate the Python expression shown on the left, math.log(x + math.sqrt(x*x + 1)), into log(x + sqrt(x*x + 1)), an expression in Odyssey's input language. This operation works for all input and output languages the LLM understands, immediately bringing Odyssey's analyses to a variety of languages that use floating-point operations. The user's work in Odyssey can be preserved in the annotation state.
• Figure 2: The LM Unit Test annotation type lets users ask yes-or-no questions about the code and suggests changes if the answer is "no." Suggestions can be applied to the buffer. The question stays with the code and can be asked again after the code is updated. This annotation type was generated in a few minutes by an LLM (see Appendix \ref{['sec:show-debugged-example-prompt']}).
• Figure 3: The document-external annotations enabled by our system could facilitate the separation of documents into layers that help with specific tasks, such as instrumenting code, without cluttering the main document.
• Figure 4: An early prototype of our system was fully editor-independent and ran in a browser. It featured (1) a document view to allow users to select annotation targets, (2) an annotation view with (user-irrelevant) configuration and document attachment information, and (3) a list of annotations. Even after explaining the system's function, users struggled to imagine workflows involving file annotations hosted in a browser separate from their editor.