Files

.github
data
images
- all-attention.png
- attention-on-attention.png
- cosine-sim-attention.png
- deepnorm.png
- dynamic-pos-bias-linear.png
- dynamic-pos-bias-log.png
- dynamic-pos-bias-sinusoidal.png
- dynamic-pos-bias.png
- enhanced-recurrence.png
- fcm.png
- ffglu.png
- flash-attention.png
- gate_values.png
- gating.png
- length-extrapolation-scale.png
- macaron-1.png
- macaron-2.png
- memory-transformer.png
- normformer.png
- pia.png
- qknorm-analysis.png
- resi_dual.png
- residual_attn.png
- rezero.png
- rotary.png
- sandwich-2.png
- sandwich.png
- sandwich_norm.png
- scalenorm.png
- talking-heads.png
- topk-attention.png
- xval.png
tests
x_transformers
.gitignore
LICENSE
README.md
pyproject.toml
train_copy.py
train_enwik8.py
train_length_extrapolate.py
train_parity.py

dynamic-pos-bias-linear.png

add dynamic positional bias, for extrapolating to greater sequence le…

Mar 3, 2022

fed9085 · Mar 3, 2022