Add QLoRA on top of the default SAM model for inference

[caroteu]

Hi @anwai98, this adds QLoRA for inference as we discussed. Using only the LoRA weights from the checkpoint and everything else from micro-sam or sam. For covid_if the inference results are

dataset	modality	model	ais	single box	single point	ib	ip
covid_if	vanilla	vit_b	nan	0.269202	0.00547444	0.0890989	0.0231846
covid_if	generalist	vit_b_lm	0.112116	0.538591	0.0941166	0.808333	0.698748
covid_if	full_ft	vit_b	0.296931	0.546952	0.214309	0.735064	0.624448
covid_if	full_ft	vit_b_lm	0.318951	0.563352	0.225422	0.826157	0.746941
covid_if	lora	vit_b	0.262948	0.560446	0.146677	0.751644	0.657821
covid_if	lora	vit_b_lm	0.30409	0.609217	0.217692	0.837911	0.769527
covid_if	qlora	vit_b	0.254248	0.5	0.057334	0.140131	0.026386
covid_if	qlora	vit_b_lm	0.308042	0.605862	0.232271	0.831093	0.755402

[caroteu]

For comparison, these were our previous qlora results:

dataset	modality	model	ais	single box	single point	ib	ip
covid_if	qlora	vit_b	0.001286	0.253488	0.001961	0.076760	0.026321
covid_if	qlora	vit_b_lm	0.001118	0.360203	0.016025	0.722143	0.545648

[anwai98]

Hi @caroteu,

Thanks for verifying this! This was my assumption. And it's good that we figured this out. I'll go ahead and merge this to our other PR (to keep a track of net changes)

PS: @constantinpape The inference hypothesis I discussed with you turns out to be correct. QLoRA in most cases is at-par to LoRA (atleast for the example data above)!

[constantinpape]

This is very nice, and we can declare QLora implemented :) .