add a norm for lime

add dynamic LIMe from Gerasimov et al., making sure it is compatible …

add dynamic LIMe from Gerasimov et al., making sure it is compatible …

fix for dynamic pos bias during inference

oops

2.0.4

2.0.3

use cautious lion for parity task

demonstrate hybridization with a gru that only acts every 4 tokens ca…

move examples to root

move examples to root

release multi-latent attention

the rotateable subhead keys from MLA needs to be cached

complete multi-latent attention

prepare for decoupled rope

first test it out without rotary

some driveby cleanup

some driveby cleanup

in multi latent attention, cache the lightweight latent kv

remove resiDual, as hyperconnections is the culmination for that line…

remove resiDual, as hyperconnections is the culmination for that line…

remove some unpopular features / research, and prepare to incorporate…

allow for queries, keys, values to be derived from different combinat…

allow for queries, keys, values to be derived from different combinat…

allow each token to decide how much of input to reinject

1.44.5

fix inp_inject not being used when using in_attn_cond (#307)

if the hybrid module is an RNN, allow for folding it across the seque…

flexibly handle hybrid module outputs

1.44.1