AI capabilities can be significantly improved without expensive retraining

TL;DR

The paper introduces compute-equivalent gain (CEG) as a unified metric to compare post-training enhancements (tool-use, prompting, scaffolding, data generation, and solution selection) across AI tasks. It analyzes a representative set of enhancements, showing most yield substantial gains with fines tuning costs typically under 1% of pre-training, though inference costs can spike. The authors argue that post-training enhancements will continue to boost frontier capabilities, especially when combined, but also raise governance and safety concerns due to broader access. They provide a framework for future measurement, including limitations and directions for improved CEG estimation and policy considerations. Overall, post-training enhancements emerge as a powerful, cost-efficient route to advancing AI capabilities with significant implications for governance and safety.

Abstract

State-of-the-art AI systems can be significantly improved without expensive retraining via "post-training enhancements"-techniques applied after initial training like fine-tuning the system to use a web browser. We review recent post-training enhancements, categorizing them into five types: tool-use, prompting methods, scaffolding, solution selection, and data generation. Different enhancements improve performance on different tasks, making it hard to compare their significance. So we translate improvements from different enhancements into a common currency, the compute-equivalent gain: how much additional training compute would be needed to improve performance by the same amount as the enhancement. Our non-experimental work shows that post-training enhancements have significant benefits: most surveyed enhancements improve benchmark performance by more than a 5x increase in training compute, some by more than 20x. Post-training enhancements are relatively cheap to develop: fine-tuning costs are typically <1% of the original training cost. Governing the development of capable post-training enhancements may be challenging because frontier models could be enhanced by a wide range of actors.

Figures (26)

• Figure 1: Illustration of the compute-equivalent gain. The enhanced model has the same performance as a non-enhanced model trained with $5\times$ more compute.
• Figure 2: Distribution of CEG and additional costs of the techniques we studied. The one-time cost is given as a fraction of pre-training, the runtime cost is relative to the runtime cost without the enhancement. Enhancements without one-time cost are shown with an arrow on the y axis.
• Figure 3: Performance of WebGPT with different quantities of demonstrations used for fine-tuning.
• Figure 4: Performance of WebGPT and GPT-3 on TruthfulQA.
• Figure 5: RETRO architecture. Extracted from Borgeaud2021.