Files

 main

/

images

/

Copy path

Directory actions

More options

Directory actions

More options

Latest commit

lucidrains

xval document (#196)

Oct 19, 2023

61e8a21 · Oct 19, 2023

History

History

/

images

/

Top

Folders and files

Name		Name	Last commit message	Last commit date
parent directory ..
all-attention.png		all-attention.png	add docs for all-attention memory key/value	Nov 25, 2020
attention-on-attention.png		attention-on-attention.png	add more docs	Nov 25, 2020
cosine-sim-attention.png		cosine-sim-attention.png	add query-key normalization attention	Dec 2, 2021
deepnorm.png		deepnorm.png	add deepnorm, successfully used in a 130B parameter model out of Tsin…	Aug 7, 2022
dynamic-pos-bias-linear.png		dynamic-pos-bias-linear.png	add dynamic positional bias, for extrapolating to greater sequence le…	Mar 3, 2022
dynamic-pos-bias-log.png		dynamic-pos-bias-log.png	add dynamic positional bias, for extrapolating to greater sequence le…	Mar 3, 2022
dynamic-pos-bias-sinusoidal.png		dynamic-pos-bias-sinusoidal.png	add dynamic positional bias, for extrapolating to greater sequence le…	Mar 3, 2022
dynamic-pos-bias.png		dynamic-pos-bias.png	add dynamic positional bias, for extrapolating to greater sequence le…	Mar 3, 2022
enhanced-recurrence.png		enhanced-recurrence.png	add enhanced recurrence, from Ernie-doc paper	Oct 3, 2021
fcm.png		fcm.png	add forgetful causal mask implementation to AutoregressiveWrapper	Nov 2, 2022
ffglu.png		ffglu.png	add docs for feedforward GLU variant	Nov 25, 2020
flash-attention.png		flash-attention.png	add flash attention	May 9, 2023
gate_values.png		gate_values.png	diagram for alphafold2 attention	Jul 16, 2021
gating.png		gating.png	add diagram for gating feature	Jan 16, 2021
length-extrapolation-scale.png		length-extrapolation-scale.png	add a technique that went under my radar, since it was not published …	Jul 26, 2022
macaron-1.png		macaron-1.png	add docs for macarons	Nov 25, 2020
macaron-2.png		macaron-2.png	add docs for macarons	Nov 25, 2020
memory-transformer.png		memory-transformer.png	add docs for memory tokens	Nov 25, 2020
normformer.png		normformer.png	add some Normformer proposed changes	Oct 20, 2021
pia.png		pia.png	add position infused attention, from DETR and Shortformer papers	Dec 31, 2020
qknorm-analysis.png		qknorm-analysis.png	update so one can apply rmsnorm on the queries and keys, due to new S…	Feb 13, 2023
resi_dual.png		resi_dual.png	add ResiDual paper	May 9, 2023
residual_attn.png		residual_attn.png	add residual attention, from Realformer paper	Dec 28, 2020
rezero.png		rezero.png	add docs for rezero is all you need	Nov 25, 2020
rotary.png		rotary.png	add documentation for rotary embeddings	Apr 21, 2021
sandwich-2.png		sandwich-2.png	add docs for sandwiches	Nov 25, 2020
sandwich.png		sandwich.png	add docs for sandwiches	Nov 25, 2020
sandwich_norm.png		sandwich_norm.png	readme	Oct 19, 2021
scalenorm.png		scalenorm.png	add docs for scale norm	Nov 25, 2020
talking-heads.png		talking-heads.png	add talking heads	Nov 25, 2020
topk-attention.png		topk-attention.png	add docs for concentrated attention paper	Nov 25, 2020
xval.png		xval.png	xval document (#196)	Oct 19, 2023
View all files