Skip to content

Commit

Permalink
plug own work
Browse files Browse the repository at this point in the history
  • Loading branch information
lucidrains committed May 15, 2023
1 parent 9cd1000 commit bfaab9a
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
<img src="./MEGABYTE.png" width="450px"></img>

## MEGABYTE-pytorch
## MEGABYTE - Pytorch

Implementation of <a href="https://arxiv.org/abs/2305.07185">MEGABYTE</a>, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

<a href="https://github.com/lucidrains/simple-hierarchical-transformer">Similar independent research</a>

## Appreciation

- <a href="https://stability.ai/">Stability</a> and <a href="https://huggingface.co/">🤗 Huggingface</a> for the generous sponsorship to work on and open source cutting edge artificial intelligence research
Expand All @@ -27,6 +29,7 @@ model = MEGABYTE(
depth = (6, 4), # number of layers for global and then local
dim_head = 64, # dimension per head
heads = 8, # number of attention heads
flash_attn = True # use flash attention
)

x = torch.randint(0, 16000, (1, 1024, 4))
Expand Down

0 comments on commit bfaab9a

Please sign in to comment.