Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Focus Your Attention (with Adaptive IIR Filters)

Shahar Lutati, Itamar Zimerman, Lior Wolf

TL;DR

The paper tackles the challenge of modeling long-range dependencies with efficient computation by introducing Focus, a layer that pre-processes input with adaptive second-order IIR filters generated by a hypernetwork. This causal, chunked approach combines a global, regularized convolution path with local attention, achieving sub-quadratic time and favorable parameter efficiency. Across associative recall, language modeling, and 1-D image tasks, Focus matches or surpasses state-of-the-art results at a fraction of the compute, while ablations underscore the critical role of online adaptivity and IIR stability. The work grounds the method in control theory and links IIR filters to state-space models and diagonal RNNs, offering a principled, scalable path for long-range sequence processing in neural architectures.

Abstract

We present a new layer in which dynamic (i.e.,input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. The input is split into chunks, and the coefficients of these filters are determined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus attention on the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-the-art networks, with a fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable to layers such as Heyna, GPT2, and Mega, both with respect to the number of parameters and the obtained level of performance on multiple long-range sequence problems.

Focus Your Attention (with Adaptive IIR Filters)

TL;DR

The paper tackles the challenge of modeling long-range dependencies with efficient computation by introducing Focus, a layer that pre-processes input with adaptive second-order IIR filters generated by a hypernetwork. This causal, chunked approach combines a global, regularized convolution path with local attention, achieving sub-quadratic time and favorable parameter efficiency. Across associative recall, language modeling, and 1-D image tasks, Focus matches or surpasses state-of-the-art results at a fraction of the compute, while ablations underscore the critical role of online adaptivity and IIR stability. The work grounds the method in control theory and links IIR filters to state-space models and diagonal RNNs, offering a principled, scalable path for long-range sequence processing in neural architectures.

Abstract

We present a new layer in which dynamic (i.e.,input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. The input is split into chunks, and the coefficients of these filters are determined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus attention on the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-the-art networks, with a fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable to layers such as Heyna, GPT2, and Mega, both with respect to the number of parameters and the obtained level of performance on multiple long-range sequence problems.
Paper Structure (20 sections, 35 equations, 4 figures, 4 tables)

This paper contains 20 sections, 35 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Learnable IIR Filters
  4. Global Convolutions
  5. Long Range Transformers
  6. Hyper Networks
  7. Adaptive Filtering
  8. Method
  9. Overview
  10. The Focus Layer
  11. Analysis
  12. Experiments
  13. In-context learning
  14. Language Modeling
  15. 1-D image classification
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Focus Architecture: (a) The architecture of a single head. (b) The obtained layer. (c) The entire model. The architecture of the model and layer are defined similarly to MEGA ma2022mega. Blocks in blue are not learned, while blocks in red are learned parameters. $S2P$ (serial to parallel) and $P2S$ (parallel to serial) are the chunking and the de-chunking operations, respectively.
  • Figure 2: Filter responses for three random filters with the specific denominator structure of Eq. \ref{['eq:iir_filter_s_plane']}.
  • Figure 3: Time Complexity of the Focus layer and of Attention, increasing sequence length
  • Figure 4: Frequency Response of IIR kernels, for 1K sequence split into 18 time bins. The important key is found in the 12th time bin.