Self-Verification is All You Need To Pass The Japanese Bar Examination
Andrew Shin
TL;DR
The paper demonstrates that preserving the original exam format and employing a lightweight self-verification step allows a single fine-tuned model to surpass the passing threshold on the Japanese bar examination without altering question structure or scoring rules. It shows that decomposing questions into independent judgments or using multi-agent reasoning does not match the performance of a format-faithful, self-verifying single model under realistic exam conditions. A carefully constructed dataset spanning six years and a consistency-checking mechanism enable the model to jointly evaluate multiple propositions and maintain global coherence. The work provides dataset and code publicly, emphasizes the importance of format-aligned supervision for high-stakes professional reasoning, and suggests that simpler, well-aligned approaches can outperform more complex systems in this domain.
Abstract
Despite rapid advances in large language models (LLMs), achieving reliable performance on highly professional and structured examinations remains a significant challenge. The Japanese bar examination is a particularly demanding benchmark, requiring not only advanced legal reasoning but also strict adherence to complex answer formats that involve joint evaluation of multiple propositions. While recent studies have reported improvements by decomposing such questions into simpler true--false judgments, these approaches have not been systematically evaluated under the original exam format and scoring scheme, leaving open the question of whether they truly capture exam-level competence. In this paper, we present a self-verification model trained on a newly constructed dataset that faithfully replicates the authentic format and evaluation scale of the exam. Our model is able to exceed the official passing score when evaluated on the actual exam scale, marking the first demonstration, to our knowledge, of an LLM passing the Japanese bar examination without altering its original question structure or scoring rules. We further conduct extensive comparisons with alternative strategies, including multi-agent inference and decomposition-based supervision, and find that these methods fail to achieve comparable performance. Our results highlight the importance of format-faithful supervision and consistency verification, and suggest that carefully designed single-model approaches can outperform more complex systems in high-stakes professional reasoning tasks. Our dataset and codes are publicly available.