Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of Long Range Dependency Understanding in State Space Models

Srividya Ravikumar, Abhinav Anand, Shweta Verma, Mira Mezini

TL;DR

This work addresses the interpretability gap in structured state-space models by examining how architecture shapes long-range dependency modeling in diagonalized S4D kernels on a real-world vulnerability-detection task. It introduce a kernel-interpretability framework that analyzes S4D kernels in both time and frequency domains across six architectures using the ReVeal dataset. The findings show that standalone S4D learns mainly short-range dependencies, while incorporating a State Memory Replay (SMR) block yields long-range, low-pass kernels and the best performance (F1 = $88.03$), highlighting the value of hybrid CNN–SSM designs. The study provides actionable insights for designing more effective SSM-based models for code analysis and similar long-sequence tasks.

Abstract

Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the diagonalized state-space model (S4D) trained on a real-world task (vulnerability detection in source code). Through time and frequency domain analysis of the S4D kernel, we show that the long-range modeling capability of S4D varies significantly under different model architectures, affecting model performance. For instance, we show that the depending on the architecture, S4D kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide future work in designing better S4D-based models.

Analysis of Long Range Dependency Understanding in State Space Models

TL;DR

This work addresses the interpretability gap in structured state-space models by examining how architecture shapes long-range dependency modeling in diagonalized S4D kernels on a real-world vulnerability-detection task. It introduce a kernel-interpretability framework that analyzes S4D kernels in both time and frequency domains across six architectures using the ReVeal dataset. The findings show that standalone S4D learns mainly short-range dependencies, while incorporating a State Memory Replay (SMR) block yields long-range, low-pass kernels and the best performance (F1 = ), highlighting the value of hybrid CNN–SSM designs. The study provides actionable insights for designing more effective SSM-based models for code analysis and similar long-sequence tasks.

Abstract

Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the diagonalized state-space model (S4D) trained on a real-world task (vulnerability detection in source code). Through time and frequency domain analysis of the S4D kernel, we show that the long-range modeling capability of S4D varies significantly under different model architectures, affecting model performance. For instance, we show that the depending on the architecture, S4D kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide future work in designing better S4D-based models.
Paper Structure (10 sections, 2 equations, 4 figures, 1 table)

This paper contains 10 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Model architectures
  4. Training
  5. Kernel analysis
  6. Time-domain analysis
  7. Frequency-domain analysis
  8. RESULTS AND DISCUSSION
  9. RELATED WORK
  10. CONCLUSION

Figures (4)

  • Figure 1: Common model architecture used across all experiments. The Feature extraction block represents different feature transformation strategies across six model variants.
  • Figure 2: Feature extraction block for SMR + S4D model.
  • Figure 3: Time-domain impulse responses of S4D kernels across architectures. Green: maximum positive amplitude, red: maximum negative amplitude, black: sharp transitions.
  • Figure 4: Frequency-domain spectra of S4D kernels across architectures. Orange: dominant frequency; black: secondary peaks ($\geq$30% of dominant).