Attention Is Not What You Need
2026-10-31
![]()
The paper proposes Causal Grassmann layers, a replacement for attention in which information moves through controlled deformations of low-rank subspaces on Grassmann manifolds rather than through attention matrices. Against size-matched Transformers, the models come within 10-15% on Wikitext-2 perplexity and slightly ahead on SNLI. Keeping the computation on a finite-dimensional manifold also makes it easier to analyse geometrically than attention's unstructured tensors.
Was this useful?