Orbit: A Framework for Designing and Evaluating Multi-objective Rankers

TL;DR

Orbit addresses the challenge of designing multi-objective rankers in production by centering objectives in the design and evaluation workflow. It introduces an objective-centric framework and an interactive system that lets stakeholders explore the objective space and assess trade-offs in real time. A user study with twelve industry practitioners shows Orbit improves design-space exploration, leads to more informed decisions, and fosters deeper consideration of trade-offs. The work suggests the approach generalizes to other multi-objective ML problems and helps bridge metric-centric and example-centric mindsets, enabling participatory design and better communication among cross-functional teams.

Abstract

Machine learning in production needs to balance multiple objectives: This is particularly evident in ranking or recommendation models, where conflicting objectives such as user engagement, satisfaction, diversity, and novelty must be considered at the same time. However, designing multi-objective rankers is inherently a dynamic wicked problem -- there is no single optimal solution, and the needs evolve over time. Effective design requires collaboration between cross-functional teams and careful analysis of a wide range of information. In this work, we introduce Orbit, a conceptual framework for Objective-centric Ranker Building and Iteration. The framework places objectives at the center of the design process, to serve as boundary objects for communication and guide practitioners for design and evaluation. We implement Orbit as an interactive system, which enables stakeholders to interact with objective spaces directly and supports real-time exploration and evaluation of design trade-offs. We evaluate Orbit through a user study involving twelve industry practitioners, showing that it supports efficient design space exploration, leads to more informed decision-making, and enhances awareness of the inherent trade-offs of multiple objectives. Orbit (1) opens up new opportunities of an objective-centric design process for any multi-objective ML models, as well as (2) sheds light on future designs that push practitioners to go beyond a narrow metric-centric or example-centric mindset.

Figures (5)

• Figure 1: Orbit's conceptual model. Objectives take the central role of ranker design: They can be translated from feedback and concrete observations, serving as the bridge between different stakeholders. They can help practitioners explore different model designs. They can help conduct evaluation, by informing what metrics to design and track and providing attribution for concrete ranking results.
• Figure 2: Orbit's interface and example usage. Orbit surfaces objectives as the first class citizen in ➀ objective overview bar, and allows users to interactively ➁ inspect, edit, or create objectives. Users can ➂ specify how multiple objectives are combined and incorporated into a model, and observe the impact in real-time. Users can look at ➃ side-by-side comparison for example-level information, and ➄ tie rankings back to objectives for explanations when needed. Users can also look at ➅ metrics and ➇ slices for aggregated information, with the ability to interactively define ➆ new metrics and new slices, and ➅ inspect slices with larger metric differences. Below the interface, we demonstrate how in our running examples, stakeholders can use Orbit to translate observations to actionable feedback, explore different designs, and gather evaluation information.
• Figure 3: Taxonomy of user activities: We characterize user activities into two categories: design and evaluation. For design, we distinguish between small-step exploration (weight-tuning) and big-step exploration (others), to understand how Orbit impacts users' design exploration in more nuances. For evaluation, we further break it down into example-based and metric-based evaluations, and distinguish between standard evaluations (dataset-level metrics, provided anecdotes) and additional evaluations (others), to understand how Orbit impact users' information-seeking behaviors.
• Figure 4: Participant's sequence of distinct activities. With Orbit, participants explored more distinct trade-offs , in bigger steps , and conducted more distinct evaluation beyond standard setups ( vs. ) in a more balanced way ( vs. ). Overall participants also explored big-step changes throughout the session with Orbit (vs. mostly only did big-step changes in the beginning followed by small weight-tuning when using notebooks).
• Figure 5: Participants found side-by-side comparison and metric tracking the most important features of Orbit, followed by objective design and data slicing.