Could you provide example code for AdaLoRA finetuning decoder-only model?

[SpeeeedLee]

Feature request

The current example of AdaLoRA is on facebook/bart-base. Since AdaLoRA requires hand-crafted calculations on loss, would it be possible to provide me some hints on how can this be done when it comes to decoder-only (e.g., Llama-Instruct) LM?

Specificially, I would like to mask out the loss calculation on the instruction part or system prompt, focusing only on the assistant response.

Motivation

AdaLoRA requires hand-crafted calculations on loss, which becomes complex when desired to mask out some system/instructino tokens.

Your contribution

N.A.

[d-kleine]

As there is response yet, I am trying to answer your question how I would approach this:

Specificially, I would like to mask out the loss calculation on the instruction part or system prompt, focusing only on the assistant response.

My idea would be to work with the special tokens in the prompt. Depending on the model you are using (the prompt template varies across different model architectures), I would look for the <|assistant|> special token. As the special tokens <|user|> and <|system|> are typically before the <|assistant|> special token, my approach would be to write custom trainer class that masks the input starting with the <|assistant|> prompt and ignores everything before that special token.

[SpeeeedLee]

Thank you for your response. Yes, I have successfully written a custom code to set the labels of tokens before <|assistant|> to -100. Then following the example code provided, a custom training loop is conducted. The results are reasonable.

[BenjaminBossan]

I'm glad to hear it worked, thanks for your suggestion @d-kleine.

If there are no further questions, feel free to close the issue @SpeeeedLee.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.