Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Design and Organization of Educational Competitions with Anonymous and Real-Time Leaderboards in Academic and Industrial Settings

Serdar Kadıoğlu, Bernard Kleynhans

TL;DR

The paper addresses how to design and operate educational competitions with anonymous, near real-time leaderboards in academic and industrial settings. It proposes a two-part contribution: (i) a comprehensive competition-planning framework spanning pre-, during-, and post-competition phases, and (ii) a high-level, platform-agnostic system design that enables near real-time evaluation while preserving anonymity. Through two experience reports—Brown University and Fidelity Investments—it demonstrates practical instantiations of the framework, detailing governance, data handling, submission workflows, and leaderboard infrastructure. The work offers actionable guidelines and architectural patterns to facilitate hands-on AI education and community engagement, while highlighting privacy, IP considerations, and cross-domain applicability in both academic courses and industry programs.

Abstract

The goal of this paper is to share our experience in designing and organizing educational competitions with anonymous and (near) real-time leaderboards in both academic and industrial settings. While such competitions serve as a great educational tool and provide participants with hands-on experience, they require significant planning, technical setup, and administration from organizers. In this paper, we first outline several important areas including team registration, data access, submission systems, rules and conditions that organizers should consider when planning such events. We then present a high-level system design that can support (near) real-time evaluation of submissions to power anonymous leaderboards and provide immediate feedback for participants. Finally, we share our experience applying this abstract system in academic and industrial settings. We hope the set of guidelines and the high-level system design proposed here help others in their organization of similar educational events.

The Design and Organization of Educational Competitions with Anonymous and Real-Time Leaderboards in Academic and Industrial Settings

TL;DR

The paper addresses how to design and operate educational competitions with anonymous, near real-time leaderboards in academic and industrial settings. It proposes a two-part contribution: (i) a comprehensive competition-planning framework spanning pre-, during-, and post-competition phases, and (ii) a high-level, platform-agnostic system design that enables near real-time evaluation while preserving anonymity. Through two experience reports—Brown University and Fidelity Investments—it demonstrates practical instantiations of the framework, detailing governance, data handling, submission workflows, and leaderboard infrastructure. The work offers actionable guidelines and architectural patterns to facilitate hands-on AI education and community engagement, while highlighting privacy, IP considerations, and cross-domain applicability in both academic courses and industry programs.

Abstract

The goal of this paper is to share our experience in designing and organizing educational competitions with anonymous and (near) real-time leaderboards in both academic and industrial settings. While such competitions serve as a great educational tool and provide participants with hands-on experience, they require significant planning, technical setup, and administration from organizers. In this paper, we first outline several important areas including team registration, data access, submission systems, rules and conditions that organizers should consider when planning such events. We then present a high-level system design that can support (near) real-time evaluation of submissions to power anonymous leaderboards and provide immediate feedback for participants. Finally, we share our experience applying this abstract system in academic and industrial settings. We hope the set of guidelines and the high-level system design proposed here help others in their organization of similar educational events.
Paper Structure (12 sections, 3 figures, 1 table)

This paper contains 12 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Competition Planning
  3. Pre-competition
  4. Kick-off
  5. In-competition
  6. Post-competition
  7. System Design
  8. Experience Report: Brown University
  9. System Design
  10. Experience Report: Fidelity Investments
  11. System Design
  12. Conclusion

Figures (3)

  • Figure 1: Components of a competition across the chronological stages before, during, and after the competition.
  • Figure 2: High-level system design with public and private components.
  • Figure 3: System design for the RecSys competition @ Fidelity Investments