Module `fast_transformers.feature_maps`

Implementations of feature maps to be used with linear attention and causal linear attention.

Expand source code

#
# Copyright (c) 2020 Idiap Research Institute, http://www.idiap.ch/
# Written by Angelos Katharopoulos <angelos.katharopoulos@idiap.ch>
#

"""Implementations of feature maps to be used with linear attention and causal
linear attention."""


from .base import elu_feature_map, ActivationFunctionFeatureMap
from .fourier_features import RandomFourierFeatures, Favor, \
    SmoothedRandomFourierFeatures, GeneralizedRandomFeatures

Sub-modules

fast_transformers.feature_maps.base: Create the feature map interface and some commonly used feature maps …
fast_transformers.feature_maps.fourier_features: Implement the positive orthogonal random features from the paper "Rethinking Attention with Performers" https://arxiv.org/pdf/2009.14794.pdf and the …