Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 generate correct code for LoRaWAN-related engineering tasks

Daniel Fernandes, João P. Matos-Carvalho, Carlos M. Fernandes, Nuno Fachada

TL;DR

This work addresses the feasibility of locally executed, lightweight LLMs to generate correct Python code for UAV placement and received-power computations within LoRaWAN networks, evaluated under zero-shot prompts. By testing 16 models across three escalating prompts and using a rigorous scoring and permutation-testing framework, the study demonstrates that DeepSeek-V3 and GPT-4 perform best overall, with Phi-4 and LLaMA-3.3 providing competitive results at a much smaller footprint. The findings reveal that carefully chosen lightweight models can approach state-of-the-art performance for targeted engineering tasks, provided prompt design and domain-specific tuning are employed. This has practical implications for cost-efficient AI-assisted engineering workflows in IoT and aerial networks, while highlighting methodological considerations for robust evaluation of code-generation in constrained domains.

Abstract

This paper investigates the performance of 16 Large Language Models (LLMs) in automating LoRaWAN-related engineering tasks involving optimal placement of drones and received power calculation under progressively complex zero-shot, natural language prompts. The primary research question is whether lightweight, locally executed LLMs can generate correct Python code for these tasks. To assess this, we compared locally run models against state-of-the-art alternatives, such as GPT-4 and DeepSeek-V3, which served as reference points. By extracting and executing the Python functions generated by each model, we evaluated their outputs on a zero-to-five scale. Results show that while DeepSeek-V3 and GPT-4 consistently provided accurate solutions, certain smaller models -- particularly Phi-4 and LLaMA-3.3 -- also demonstrated strong performance, underscoring the viability of lightweight alternatives. Other models exhibited errors stemming from incomplete understanding or syntactic issues. These findings illustrate the potential of LLM-based approaches for specialized engineering applications while highlighting the need for careful model selection, rigorous prompt design, and targeted domain fine-tuning to achieve reliable outcomes.

DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 generate correct code for LoRaWAN-related engineering tasks

TL;DR

This work addresses the feasibility of locally executed, lightweight LLMs to generate correct Python code for UAV placement and received-power computations within LoRaWAN networks, evaluated under zero-shot prompts. By testing 16 models across three escalating prompts and using a rigorous scoring and permutation-testing framework, the study demonstrates that DeepSeek-V3 and GPT-4 perform best overall, with Phi-4 and LLaMA-3.3 providing competitive results at a much smaller footprint. The findings reveal that carefully chosen lightweight models can approach state-of-the-art performance for targeted engineering tasks, provided prompt design and domain-specific tuning are employed. This has practical implications for cost-efficient AI-assisted engineering workflows in IoT and aerial networks, while highlighting methodological considerations for robust evaluation of code-generation in constrained domains.

Abstract

This paper investigates the performance of 16 Large Language Models (LLMs) in automating LoRaWAN-related engineering tasks involving optimal placement of drones and received power calculation under progressively complex zero-shot, natural language prompts. The primary research question is whether lightweight, locally executed LLMs can generate correct Python code for these tasks. To assess this, we compared locally run models against state-of-the-art alternatives, such as GPT-4 and DeepSeek-V3, which served as reference points. By extracting and executing the Python functions generated by each model, we evaluated their outputs on a zero-to-five scale. Results show that while DeepSeek-V3 and GPT-4 consistently provided accurate solutions, certain smaller models -- particularly Phi-4 and LLaMA-3.3 -- also demonstrated strong performance, underscoring the viability of lightweight alternatives. Other models exhibited errors stemming from incomplete understanding or syntactic issues. These findings illustrate the potential of LLM-based approaches for specialized engineering applications while highlighting the need for careful model selection, rigorous prompt design, and targeted domain fine-tuning to achieve reliable outcomes.

Paper Structure

This paper contains 15 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. LLMs for Human--UAV Interaction
  4. Code Generation with LLMs
  5. Prompt Design
  6. Materials and Methods
  7. Theoretical Context
  8. Scenarios and Prompts
  9. LLMs Considered
  10. Implementation
  11. Experimental Setup
  12. Results
  13. Discussion
  14. Limitations
  15. Conclusions

Figures (2)

  • Figure 1: Validation pipeline for the results of LLMs under study.
  • Figure 2: Pairwise significance heatmap of model performance comparisons for the three prompts across temperatures. Each colored block represents the $p$-value of a one-sided, rank-based stratified permutation test between two models (model in row vs. model in column) for a given temperature. Cells are colored based on statistical significance after Benjamini--Hochberg FDR multiple testing correction: dark green indicates a significant advantage of the model in the row against the model in the column ($p < 0.01$), light green indicates moderate significant advantage ($p < 0.05$), and light gray denotes no significant difference. Temperatures for online models, deepseek-v3 and gpt-4-0613, are twice the displayed values.