Generating Wikipedia by Summarizing Long Sequences

memory-compressed-attention

CBAM: Convolutional Block Attention Module

attention-module

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks

set_transformer

CCNet: Criss-Cross Attention for Semantic Segmentation

CCNet

Efficient Attention: Attention with Linear Complexities

efficient-attention

Star-Transformer

fastNLP

GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond

GCNet

Generating Long Sequences with Sparse Transformers

DeepSpeed

SCRAM: Spatially Coherent Randomized Attention Maps

:heavy_check_mark:

Interlaced Sparse Self-Attention for Semantic Segmentation

IN_PAPER

Permutohedral Attention Module for Efficient Non-Local Neural Networks

Permutohedral_attention_module

Large Memory Layers with Product Keys

XLM

Expectation-Maximization Attention Networks for Semantic Segmentation

EMANet

BP-Transformer: Modelling Long-Range Context via Binary Partitioning

BPT

Compressive Transformers for Long-Range Sequence Modelling

compressive-transformer-pytorch

Axial Attention in Multidimensional Transformers

axial-attention

Reformer: The Efficient Transformer

trax

Sparse Sinkhorn Attention

sinkhorn-transformer

Transformer on a Diet

transformer-on-diet

Time-aware Large Kernel Convolutions

TaLKConvolutions

SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection

:heavy_check_mark:

Efficient Content-Based Sparse Attention with Routing Transformers

routing-transformer

Neural Architecture Search for Lightweight Non-Local Networks

AutoNL

Longformer: The Long-Document Transformer

longformer

ETC: Encoding Long and Structured Inputs in Transformers

EXPANDcombines global attention (star transformer with multiple global tokens) with local attention

Multi-scale Transformer Language Models

IN_PAPER

Synthesizer: Rethinking Self-Attention in Transformer Models

Synthesizer-Rethinking-Self-Attention-Transformer-Models

Jukebox: A Generative Model for Music

jukebox

Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers

:heavy_check_mark:

GMAT: Global Memory Augmentation for Transformers

gmat

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

fast-transformers

Linformer: Self-Attention with Linear Complexity

linformer-pytorch

Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

google-research

Kronecker Attention Networks

kronecker-attention-pytorch

Real-time Semantic Segmentation with Fast Attention

EXPANDl2_norm(q)*(l2_norm(k)*v)

Fast Transformers with Clustered Attention

fast-transformers

Big Bird: Transformers for Longer Sequences

DeepSpeed

Tensor Low-Rank Reconstruction for Semantic Segmentation

EXPANDdecompose the full attention tensor into rank one tensors (CP decomposition)

Looking for change? Roll the Dice and demand Attention

IN_PAPER

Rethinking Attention with Performers

google-research

Memformer: The Memory-Augmented Transformer

memformer

SMYRF: Efficient Attention using Asymmetric Clustering

smyrf

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Informer2020

Sub-Linear Memory: How to Make Performers SLiM

google-research

Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

Nystromformer

Linear Transformers Are Secretly Fast Weight Memory Systems

fast-weight-transformers

LambdaNetworks: Modeling Long-Range Interactions Without Attention

lambda-networks

Random Feature Attention

:heavy_check_mark:

A Survey of Long-Term Context in Transformers

Efficient Transformers: A Survey

Long Range Arena: A Benchmark for Efficient Transformers