Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Fatih Uenal

Abstract

While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.

Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

Abstract

While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.
Paper Structure (39 sections, 1 equation, 5 figures, 7 tables)

This paper contains 39 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Benchmarks for Legal Tasks
  4. Swiss Legal NLP
  5. LLM-as-Judge Methodology
  6. AI Compliance and Regulatory Frameworks
  7. Methodology
  8. Dataset Construction and Composition
  9. Model Selection
  10. Structured Scoring Framework
  11. Multi-Judge Evaluation Panel
  12. Statistical Analysis
  13. Results
  14. Overall Rankings and Tier Structure
  15. Per-Domain Analysis
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Inter-rater reliability improvement from grade-mode to structured scoring. Dashed lines indicate Landis & Koch interpretation thresholds. Structured scoring moves the panel average from moderate ($\kappa = 0.512$) to substantial ($\kappa = 0.605$) agreement.
  • Figure 2: Overall model ranking by correct rate on 395 items. Colors indicate tier: dark (A), medium (B), light (C).
  • Figure 3: Correct rate by regulatory domain. Legal-CH is much easier than FINMA and EFK (Reg. QA) across all models. Note: EFK results are confounded with the Regulatory Q&A task type (all 48 EFK items belong to this single task type).
  • Figure 4: Heatmap of correct rate (%) by model and task type. Case analysis and legal translation are easiest; regulatory Q&A and gap analysis are hardest.
  • Figure 5: Correct rate by language (descriptive only). Italian items yield higher C% across all models, but this is heavily confounded by compositional bias: 30.9% of Italian items are case analysis or legal translation vs. 6.7% for German. See Section \ref{['sec:discussion']} and Table \ref{['tab:crosstab']}.