Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Accuracy Optimization: Computer Vision Losses for Large Language Model Fine-Tuning

Daniele Rege Cambrin, Giuseppe Gallipoli, Irene Benedetto, Luca Cagliero, Paolo Garza

TL;DR

This work tackles the efficiency limits of fine-tuning large language models by repurposing segmentation-style losses from computer vision for NLP tasks. By combining these losses with a light LoRA-based fine-tuning regime, the authors show that task-adapted losses such as Focal and Lovász can significantly outperform standard Cross Entropy on Math Word Problems and QA, achieving up to a +42% gain in exact-match without additional data or human feedback. They also demonstrate improved quality of intermediate reasoning steps, with IoU-aligned objectives correlating with reasoning metrics and reducing error types like missing steps. The findings suggest a scalable, data-efficient path for improving LLM performance across structured reasoning tasks, with practical implications for accessible model fine-tuning and broader deployment.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance across various tasks. However, current training approaches combine standard cross-entropy loss with extensive data, human feedback, or ad hoc methods to enhance performance. These solutions are often not scalable or feasible due to their associated costs, complexity, or resource requirements. This study investigates the use of established semantic segmentation loss functions in natural language generation to create a versatile, practical, and scalable solution for fine-tuning different architectures. We evaluate their effectiveness in solving Math Word Problems and question answering across different models of varying sizes. For the analyzed tasks, we found that the traditional Cross-Entropy loss represents a sub-optimal choice, while models trained to minimize alternative (task-dependent) losses, such as Focal or Lovász, achieve a mean improvement of +42% on exact match without requiring additional data or human feedback. These findings suggest a promising pathway for more efficient and accessible training processes.

Beyond Accuracy Optimization: Computer Vision Losses for Large Language Model Fine-Tuning

TL;DR

This work tackles the efficiency limits of fine-tuning large language models by repurposing segmentation-style losses from computer vision for NLP tasks. By combining these losses with a light LoRA-based fine-tuning regime, the authors show that task-adapted losses such as Focal and Lovász can significantly outperform standard Cross Entropy on Math Word Problems and QA, achieving up to a +42% gain in exact-match without additional data or human feedback. They also demonstrate improved quality of intermediate reasoning steps, with IoU-aligned objectives correlating with reasoning metrics and reducing error types like missing steps. The findings suggest a scalable, data-efficient path for improving LLM performance across structured reasoning tasks, with practical implications for accessible model fine-tuning and broader deployment.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance across various tasks. However, current training approaches combine standard cross-entropy loss with extensive data, human feedback, or ad hoc methods to enhance performance. These solutions are often not scalable or feasible due to their associated costs, complexity, or resource requirements. This study investigates the use of established semantic segmentation loss functions in natural language generation to create a versatile, practical, and scalable solution for fine-tuning different architectures. We evaluate their effectiveness in solving Math Word Problems and question answering across different models of varying sizes. For the analyzed tasks, we found that the traditional Cross-Entropy loss represents a sub-optimal choice, while models trained to minimize alternative (task-dependent) losses, such as Focal or Lovász, achieve a mean improvement of +42% on exact match without requiring additional data or human feedback. These findings suggest a promising pathway for more efficient and accessible training processes.
Paper Structure (54 sections, 7 equations, 3 figures, 12 tables)

This paper contains 54 sections, 7 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Training methods for Large Language Models
  4. Pre-Training.
  5. Supervised Fine-tuning and Instruction Tuning.
  6. Human Feedback.
  7. Loss functions beyond Cross-Entropy
  8. Methodology
  9. Distribution-based losses
  10. Cross Entropy Loss.
  11. Focal Loss.
  12. Region-based losses
  13. Dice Loss.
  14. Generalized Dice Loss.
  15. Lovász Loss.
  16. ...and 39 more sections

Figures (3)

  • Figure 1: Percentage of improvement using the best loss (among Focal, Lovász, Generalized Dice, and Self-Adjusting Dice) for the task with Cross-Entropy compared to model fine-tuned with Cross-Entropy only.
  • Figure 2: A graphical sketch of how the combined loss is applied to instruction $I$ and answer $A$. ITs are instruction tokens, ATs are answer tokens.
  • Figure 3: Kernel Density Estimation in log scale for token distributions in GSM8K, MathQA, OpenBookQA, and HellaSwag datasets.