An Analysis of Embedding Layers and Similarity Scores using Siamese Neural Networks

TL;DR

The paper investigates embedding-layer algorithms (BERT, PaLM, OpenAI) and their sentence-similarity scores in the context of medical question pairs, incorporating a Siamese Neural Network to assess potential performance gains. It compares cosine baselines to model-specific similarity metrics and evaluates environmental impact by measuring carbon footprint per training epoch and in total. Key findings show OpenAI delivers the highest accuracy but at substantial carbon cost, PaLM offers a strong accuracy–energy footprint balance with the lowest total emissions, and BERT provides solid performance with the lowest per-epoch emissions. The work highlights important trade-offs between accuracy, stability, and sustainability in embedding choices for large language model applications, with practical implications for scalable and eco-friendly deployment.

Abstract

Large Lanugage Models (LLMs) are gaining increasing popularity in a variety of use cases, from language understanding and writing to assistance in application development. One of the most important aspects for optimal funcionality of LLMs is embedding layers. Word embeddings are distributed representations of words in a continuous vector space. In the context of LLMs, words or tokens from the input text are transformed into high-dimensional vectors using unique algorithms specific to the model. Our research examines the embedding algorithms from leading companies in the industry, such as OpenAI, Google's PaLM, and BERT. Using medical data, we have analyzed similarity scores of each embedding layer, observing differences in performance among each algorithm. To enhance each model and provide an additional encoding layer, we also implemented Siamese Neural Networks. After observing changes in performance with the addition of the model, we measured the carbon footage per epoch of training. The carbon footprint associated with large language models (LLMs) is a significant concern, and should be taken into consideration when selecting algorithms for a variety of use cases. Overall, our research compared the accuracy different, leading embedding algorithms and their carbon footage, allowing for a holistic review of each embedding algorithm.

Figures (3)

• Figure 1: Model of the Siamese Neural Network used
• Figure 2: Box Chart showing average accuracy of similarity scores
• Figure 3: Box Chart showing average accuracy of the base Siamese Neural Network