Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
About
State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter {\Delta}. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Commonsense Reasoning | HellaSwag | -- | 1891 | |
| Commonsense Reasoning | WinoGrande | Accuracy56.35 | 1085 | |
| Commonsense Reasoning | PIQA | Accuracy71.33 | 751 | |
| Language Modeling | WikiText | PPL20.41 | 732 | |
| Commonsense Reasoning | HellaSwag | HellaSwag Accuracy51.85 | 350 | |
| Language Modeling | LAMBADA | Accuracy41.18 | 268 | |
| Common Sense Reasoning | BoolQ | Accuracy60.73 | 212 | |
| Commonsense Reasoning | ARC Challenge | Accuracy34.81 | 190 | |
| Language Modeling | LAMBADA | Perplexity21.74 | 150 | |
| Common Sense Reasoning | ARC Easy | ARC (easy) Accuracy68.48 | 72 |