Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Nova 2.0 Lite model under Amazon's Frontier Model Safety Framework

Satyapriya Krishna, Matteo Memelli, Tong Wang, Abhinav Mohanty, Claire O'Brien Rajkumar, Payal Motwani, Rahul Gupta, Spyros Matsoukas

TL;DR

This work evaluates Nova 2.0 Lite against Amazon's Frontier Model Safety Framework, using automated benchmarks and human-centric red-teaming to assess risks in CBRN proliferation, Offensive Cyber Operations, and Automated AI R&D. Across domains, the model shows knowledge gains over predecessors but remains within release thresholds, aided by a layered mitigation stack including dynamic content filters and monitoring. Independent audits and uplift studies corroborate safety for public deployment while highlighting specific areas (e.g., radiological risk uplift) that warrant ongoing safeguards. By formalizing the evaluation protocol and sharing empirical risk signals, the paper provides a transparent, auditable blueprint for frontier-model safety assessments. The findings support responsible deployment and offer a reference template for cross-organizational safety audits in frontier AI systems.

Abstract

Amazon published its Frontier Model Safety Framework (FMSF) as part of the Paris AI summit, following which we presented a report on Amazon's Premier model. In this report, we present an evaluation of Nova 2.0 Lite. Nova 2.0 Lite was made generally available from amongst the Nova 2.0 series and is one of its most capable reasoning models. The model processes text, images, and video with a context length of up to 1M tokens, enabling analysis of large codebases, documents, and videos in a single prompt. We present a comprehensive evaluation of Nova 2.0 Lite's critical risk profile under the FMSF. Evaluations target three high-risk domains-Chemical, Biological, Radiological and Nuclear (CBRN), Offensive Cyber Operations, and Automated AI R&D-and combine automated benchmarks, expert red-teaming, and uplift studies to determine whether the model exceeds release thresholds. We summarize our methodology and report core findings. We will continue to enhance our safety evaluation and mitigation pipelines as new risks and capabilities associated with frontier models are identified.

Evaluating Nova 2.0 Lite model under Amazon's Frontier Model Safety Framework

TL;DR

This work evaluates Nova 2.0 Lite against Amazon's Frontier Model Safety Framework, using automated benchmarks and human-centric red-teaming to assess risks in CBRN proliferation, Offensive Cyber Operations, and Automated AI R&D. Across domains, the model shows knowledge gains over predecessors but remains within release thresholds, aided by a layered mitigation stack including dynamic content filters and monitoring. Independent audits and uplift studies corroborate safety for public deployment while highlighting specific areas (e.g., radiological risk uplift) that warrant ongoing safeguards. By formalizing the evaluation protocol and sharing empirical risk signals, the paper provides a transparent, auditable blueprint for frontier-model safety assessments. The findings support responsible deployment and offer a reference template for cross-organizational safety audits in frontier AI systems.

Abstract

Amazon published its Frontier Model Safety Framework (FMSF) as part of the Paris AI summit, following which we presented a report on Amazon's Premier model. In this report, we present an evaluation of Nova 2.0 Lite. Nova 2.0 Lite was made generally available from amongst the Nova 2.0 series and is one of its most capable reasoning models. The model processes text, images, and video with a context length of up to 1M tokens, enabling analysis of large codebases, documents, and videos in a single prompt. We present a comprehensive evaluation of Nova 2.0 Lite's critical risk profile under the FMSF. Evaluations target three high-risk domains-Chemical, Biological, Radiological and Nuclear (CBRN), Offensive Cyber Operations, and Automated AI R&D-and combine automated benchmarks, expert red-teaming, and uplift studies to determine whether the model exceeds release thresholds. We summarize our methodology and report core findings. We will continue to enhance our safety evaluation and mitigation pipelines as new risks and capabilities associated with frontier models are identified.
Paper Structure (15 sections, 3 figures)

This paper contains 15 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Chemical, Biological, Radiological, and Nuclear (CBRN) Weapons Proliferation
  3. Automated Benchmarks
  4. Human-Centric Risk Evaluation
  5. Offensive Cyber Operations
  6. Automated Benchmarks
  7. Qualitative Analysis of Model Traces on CTF Challenges
  8. Human-Centric Risk Evaluation
  9. Automated AI R&D
  10. Automated Benchmarks
  11. Code-Intensive Research Tasks.
  12. Assessment.
  13. Human-Centric Risk Evaluation
  14. Conclusion
  15. Acknowledgements

Figures (3)

  • Figure 1: CBRN automated benchmark results comparing models across four datasets. For Nova 2.0 Lite, we present the best result among all the reasoning modes (low','medium', and 'high').
  • Figure 2: Offensive Cyber Security Knowledge Benchmarks comparing Nova 2.0 Lite against previous generation baselines. For Nova 2.0 Lite, we present the best result among all the reasoning modes ('low', 'medium', and 'high').
  • Figure 3: Cybench Benchmark Performance Comparison. Nova 2.0 Lite demonstrates a 7.5% uplift in guided mode compared to previous versions, showing strong alignment with foundational security tasks.