Regarding the baseline performance and hyperparameters

[Arnav0400]

Hello Authors,

Thanks for your work and this codebase, I had some questions regarding your implementation and hyperparameters of baselines like LoRA.
Specifically -

1. What is the rank in LoRA, which layers have a LoRA branch (Attention or MLP), does adding LoRA to all linear layers improve the performance (as shown in recent works), any ablations done on basic LoRA for visual tasks?
2. Since LoRA is completely re-parameterizable post-tuning, it should be included in NO extra structure section.

I am planning on extending your work for much smaller models and hence any insights would be really appreciated.

Regards,
Arnav