kapynResearch

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

State Space Models (SSMs) are a promising alternative to Transformers, offering constant memory and linear compute for sequence modeling. However, they still trail Transformers on in‑context learning and precise retrieval tasks, limiting large‑scale language model adoption. This paper shows that the gating mechanism largely determines SSMs’ success and failure in these domains.

Apple ML Research·Nov 10, 2026

Opening Kapyn…