How to determine the parameters for the t5_gemm file?

[eycheung]

I'm trying to run the FasterTransformer code on our own T5 model. When building the t5_gemm file, I'm not really sure what some of the input parameters are referencing to. Can someone give me some pointers?

• encoder_size_per_head: What is the size here referring to?
• encoder_inter_size: What is inter_size? I've seen references suggesting to use 4 * head_num * size_per_head., but I'm not sure if this is something I need to infer from my pretrained model or if this rule of thumb generally works and is something that can be tuned for more optimizations.

Thanks for the help in advance!

[byshiue]

encoder_size_per_head: What is the size here referring to?

It is size_per_head in encoder, or called by hidden_size in some cases.

encoder_inter_size: What is inter_size? I've seen references suggesting to use 4 * head_num * size_per_head., but I'm not sure if this is something I need to infer from my pretrained model or if this rule of thumb generally works and is something that can be tuned for more optimizations.

The intermediate size of Feed Forward Network.