How to use a LoRA.

[Herman5555]

Does anyone know if you can use a LoRA?, unfortunately there are sparse guides and conflicting information. What is the easiest way to apply a LoRA to a GGUF quantized model?. Any helpful information is much appreciated.

[LostRuins]

The best way is to merge it into the base model externally, before converting it to GGUF, and then you convert the result into a pure new GGUF. You will have best quality and compatibility that way.

[StripedPuppy]

What's the difference between LORA and LORA base in the configuration? I run out of pinned memory when loading into LORA, but I'd imagine that's because it's trying to load both a model and LORA.

LORA base works, though.

[Herman5555]

The best way is to merge it into the base model externally, before converting it to GGUF, and then you convert the result into a pure new GGUF. You will have best quality and compatibility that way.

Thanks for your answer, much appreciated.

[deathtome1998]

what I don't know though is how to load a Lora in the first place. I have a folder with a Lora that should have whatever the file is I need in there, but I have no clue which file in the main folder or which of the three checkpoint subfolders would have the file I need. I already have a GGUF model I have used in the past. but now that I want to add this Lora to it I'm at a loss. any help is greatly appreciated.

[gustrd]

I believe you will need to use the Transformers library from HuggingFace at Python to merge de weights, then convert to GGUF.

Here is an example of how the merge is done: https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/3