Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence

Qiushi Sun, Jingyang Gong, Yang Liu, Qiaosheng Chen, Lei Li, Kai Chen, Qipeng Guo, Ben Kao, Fei Yuan

TL;DR

JanusCoder presents a unified visual-programmatic interface for multimodal code intelligence, addressing the scarcity of high-quality multimodal data with a complete data-synthesis toolkit that yields the JanusCode-800K corpus. The authors train JanusCoder and JanusCoderV on this corpus, achieving strong performance across unimodal and multimodal tasks and approaching or surpassing leading commercial models on several benchmarks. The work demonstrates the value of cross-domain data synergies and a VLM-guided reward model to ensure data quality, enabling a scalable, open-source foundation for future research and applications in chart-to-code, WebUI generation, and dynamic visualizations. By integrating diverse visual domains (charts, animations, interactive UIs) with code, JanusCoder advances practical multimodal code generation, editing, and understanding with broad potential impact in data visualization, education, and software tooling.

Abstract

The scope of neural code intelligence is rapidly expanding beyond text-based source code to encompass the rich visual outputs that programs generate. This visual dimension is critical for advanced applications like flexible content generation and precise, program-driven editing of visualizations. However, progress has been impeded by the scarcity of high-quality multimodal code data, a bottleneck stemming from challenges in synthesis and quality assessment. To address these challenges, we make contributions from both a data and modeling perspective. We first introduce a complete synthesis toolkit that leverages reciprocal synergies between data modalities to efficiently produce a large-scale, high-quality corpus spanning from standard charts to complex interactive web UIs and code-driven animations. Leveraging this toolkit, we construct JanusCode-800K, the largest multimodal code corpus to date. This powers the training of our models, JanusCoder and JanusCoderV, which establish a visual-programmatic interface for generating code from textual instructions, visual inputs, or a combination of both. Our unified model is a departure from existing approaches that build specialized models for isolated tasks. Extensive experiments on both text-centric and vision-centric coding tasks demonstrate the superior performance of the JanusCoder series, with our 7B to 14B scale models approaching or even exceeding the performance of commercial models. Furthermore, extensive analysis provides key insights into harmonizing programmatic logic with its visual expression. Our code and checkpoints will are available at https://github.com/InternLM/JanusCoder.

JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence

TL;DR

JanusCoder presents a unified visual-programmatic interface for multimodal code intelligence, addressing the scarcity of high-quality multimodal data with a complete data-synthesis toolkit that yields the JanusCode-800K corpus. The authors train JanusCoder and JanusCoderV on this corpus, achieving strong performance across unimodal and multimodal tasks and approaching or surpassing leading commercial models on several benchmarks. The work demonstrates the value of cross-domain data synergies and a VLM-guided reward model to ensure data quality, enabling a scalable, open-source foundation for future research and applications in chart-to-code, WebUI generation, and dynamic visualizations. By integrating diverse visual domains (charts, animations, interactive UIs) with code, JanusCoder advances practical multimodal code generation, editing, and understanding with broad potential impact in data visualization, education, and software tooling.

Abstract

The scope of neural code intelligence is rapidly expanding beyond text-based source code to encompass the rich visual outputs that programs generate. This visual dimension is critical for advanced applications like flexible content generation and precise, program-driven editing of visualizations. However, progress has been impeded by the scarcity of high-quality multimodal code data, a bottleneck stemming from challenges in synthesis and quality assessment. To address these challenges, we make contributions from both a data and modeling perspective. We first introduce a complete synthesis toolkit that leverages reciprocal synergies between data modalities to efficiently produce a large-scale, high-quality corpus spanning from standard charts to complex interactive web UIs and code-driven animations. Leveraging this toolkit, we construct JanusCode-800K, the largest multimodal code corpus to date. This powers the training of our models, JanusCoder and JanusCoderV, which establish a visual-programmatic interface for generating code from textual instructions, visual inputs, or a combination of both. Our unified model is a departure from existing approaches that build specialized models for isolated tasks. Extensive experiments on both text-centric and vision-centric coding tasks demonstrate the superior performance of the JanusCoder series, with our 7B to 14B scale models approaching or even exceeding the performance of commercial models. Furthermore, extensive analysis provides key insights into harmonizing programmatic logic with its visual expression. Our code and checkpoints will are available at https://github.com/InternLM/JanusCoder.

Paper Structure

This paper contains 65 sections, 1 equation, 11 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Code Generation for Visual Interfaces.
  4. Visually-Grounded Code Generation and Understanding.
  5. Method
  6. Data Collection
  7. Data Curation
  8. Guided Evolution.
  9. Re-Contextualization.
  10. Reverse Instruction.
  11. Bidirectional Translation.
  12. Cross-Domain Synergies
  13. Data Quality Control
  14. JanusCode-800K
  15. DTVBench
  16. ...and 50 more sections

Figures (11)

  • Figure 1: JanusCoder is a suite of models that establishes a unified visual-programmatic interface, advancing multimodal code intelligence. It supports diverse tasks by combining code with visual content generation, editing, and interpretation in a unified manner.
  • Figure 2: An overview of our toolkit for curating JanusCode-800K, which integrates heterogeneous data sourcing, multi-strategy synthesis and curation, and LLM/VLM-based reward modeling after execution checks.
  • Figure 2: Statistics of JanusCode-800K.
  • Figure 3: Distribution of JanusCode-800K.
  • Figure 4: Effectiveness on different model backbones
  • ...and 6 more figures