Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

About

State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter {\Delta}. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.

Yehjin Shin, Seojin Kim, Noseong Park• 2026

Related benchmarks

TaskDatasetResultRank
Commonsense ReasoningHellaSwag--
1896
Commonsense ReasoningWinoGrande
Accuracy56.35
1581
Commonsense ReasoningHellaSwag
HellaSwag Accuracy51.85
897
Commonsense ReasoningPIQA
Accuracy71.33
757
Language ModelingWikiText
PPL20.41
740
Language ModelingLAMBADA
Accuracy41.18
412
Common Sense ReasoningBoolQ
Accuracy60.73
280
Commonsense ReasoningARC Challenge
Accuracy34.81
259
Language ModelingLAMBADA
Perplexity21.74
254
Common Sense ReasoningARC Easy
ARC (easy) Accuracy68.48
149
Showing 10 of 16 rows

Other info

Follow for update