New inpainting checkpoint from RunwayML

[monophthongal]

RunwayML has published a new inpainting model with funding from Stability. Appears to be a standard SD compatible checkpoint, but the GitHub mentions additional input channels:

Resumed from sd-v1-2.ckpt. First 595k steps regular training, then 440k steps of inpainting training at resolution 512x512 on "laion-aesthetics v2 5+" and 10% dropping of the text-conditioning to improve classifier-free guidance sampling. For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint. During training, we generate synthetic masks and in 25% mask everything.

Will this work with the webui?

See here:

https://twitter.com/runwayml/status/1582441479384858626

GitHub:

https://github.com/runwayml/stable-diffusion

[ClashSAN]

Dunno, just trying it

[Pdonor]

it looks GREAT here. https://github.com/runwayml/stable-diffusion
Can't wait for somebody to adapt this for this GUI, looks like it should be easy- download the checkpoint and (make a few changes? )

[monophthongal]

I spent a few minutes looking through their code and it looks like the biggest difference is a dozen or so lines of added code in the DDIM sampler.

[1blackbar]

in their demo on website the inpainting thing is pretty amazing, it just does what you describe to it https://app.runwayml.com/

[monophthongal]

Apparently this checkpoint was finetuned from SD 1.5:

https://github.com/runwayml/stable-diffusion/commit/4eb9692d3fd0c71260f2bc5b2a7e858233737f7f#diff-a4d20bf92713d027e2f325fa6f6ec01d8ce3e39dc9f381afada6d5a1fa8bd0a2R107

[aniketgore]

So we may finally have an unofficial 1.5 checkpoint through it.

[Ehplodor]

No, sorry. from their Github :
"sd-v1-5-inpainting.ckpt: Resumed from sd-v1-2.ckpt. First 595k steps regular training, then 440k steps of inpainting training at resolution 512x512 on "laion-aesthetics v2 5+" and 10% dropping of the text-conditioning to improve classifier-free guidance sampling. For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint. During training, we generate synthetic masks and in 25% mask everything."

So they actually "invented" 1.5

[aniketgore]

It doesn't work as is in the UI.

\venv\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

[cibernicola]

Link is working, buut chckpoint doesn't load correctly:
Loading weights [3e16efc8] from L:\StableDiffussionWebUI\models\Stable-diffusion\sd-v1-5-inpainting.ckpt Global Step: 440000 Traceback (most recent call last): File "L:\StableDiffussionWebUI\venv\lib\site-packages\gradio\routes.py", line 275, in run_predict output = await app.blocks.process_api( File "L:\StableDiffussionWebUI\venv\lib\site-packages\gradio\blocks.py", line 787, in process_api result = await self.call_function(fn_index, inputs, iterator) File "L:\StableDiffussionWebUI\venv\lib\site-packages\gradio\blocks.py", line 694, in call_function prediction = await anyio.to_thread.run_sync( File "L:\StableDiffussionWebUI\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "L:\StableDiffussionWebUI\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "L:\StableDiffussionWebUI\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run result = context.run(func, *args) File "L:\StableDiffussionWebUI\modules\ui.py", line 1633, in <lambda> fn=lambda value, k=k: run_settings_single(value, key=k), File "L:\StableDiffussionWebUI\modules\ui.py", line 1488, in run_settings_single opts.data_labels[key].onchange() File "L:\StableDiffussionWebUI\webui.py", line 40, in f res = func(*args, **kwargs) File "L:\StableDiffussionWebUI\webui.py", line 85, in <lambda> shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights(shared.sd_model))) File "L:\StableDiffussionWebUI\modules\sd_models.py", line 249, in reload_model_weights load_model_weights(sd_model, checkpoint_info) File "L:\StableDiffussionWebUI\modules\sd_models.py", line 169, in load_model_weights missing, extra = model.load_state_dict(sd, strict=False) File "L:\StableDiffussionWebUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for LatentDiffusion: size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

[rytt0001]

too bad "Access to model runwayml/stable-diffusion-inpainting is restricted and you are not in the authorized list. Visit https://huggingface.co/runwayml/stable-diffusion-inpainting to ask for access."

I know that someone downloaded it and is sharing it on google drive, if you really want to dl it at your own risk search a post talking about it on reddit.

[rytt0001]

Link is working, buut chckpoint doesn't load correctly: Loading weights [3e16efc8] from L:\StableDiffussionWebUI\models\Stable-diffusion\sd-v1-5-inpainting.ckpt Global Step: 440000 Traceback (most recent call last): File "L:\StableDiffussionWebUI\venv\lib\site-packages\gradio\routes.py", line 275, in run_predict output = await app.blocks.process_api( File "L:\StableDiffussionWebUI\venv\lib\site-packages\gradio\blocks.py", line 787, in process_api result = await self.call_function(fn_index, inputs, iterator) File "L:\StableDiffussionWebUI\venv\lib\site-packages\gradio\blocks.py", line 694, in call_function prediction = await anyio.to_thread.run_sync( File "L:\StableDiffussionWebUI\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "L:\StableDiffussionWebUI\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "L:\StableDiffussionWebUI\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run result = context.run(func, *args) File "L:\StableDiffussionWebUI\modules\ui.py", line 1633, in <lambda> fn=lambda value, k=k: run_settings_single(value, key=k), File "L:\StableDiffussionWebUI\modules\ui.py", line 1488, in run_settings_single opts.data_labels[key].onchange() File "L:\StableDiffussionWebUI\webui.py", line 40, in f res = func(*args, **kwargs) File "L:\StableDiffussionWebUI\webui.py", line 85, in <lambda> shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights(shared.sd_model))) File "L:\StableDiffussionWebUI\modules\sd_models.py", line 249, in reload_model_weights load_model_weights(sd_model, checkpoint_info) File "L:\StableDiffussionWebUI\modules\sd_models.py", line 169, in load_model_weights missing, extra = model.load_state_dict(sd, strict=False) File "L:\StableDiffussionWebUI\venv\lib\site-packages\torch\nn\modules\module.py", line 1604, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for LatentDiffusion: size mismatch for model.diffusion_model.input_blocks.0.0.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

it is a normal error the stated that when training the model For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself)
so the model is not currently compatible with the standard CompVis model arch

[Ehplodor]

too bad "Access to model runwayml/stable-diffusion-inpainting is restricted and you are not in the authorized list. Visit https://huggingface.co/runwayml/stable-diffusion-inpainting to ask for access."

I know that someone downloaded it and is sharing it on google drive, if you really want to dl it at your own risk search a post talking about it on reddit.

Sry just had to accept licence ==> currently downloading

[cibernicola]

Yup. only need to log in, that's all.

[AmericanPresidentJimmyCarter]

Implemented for k_diffusion samplers here, see test.py for use: https://github.com/AmericanPresidentJimmyCarter/stable-diffusion/tree/inpainting-model

img2img does not work and multi-cond conditioning does not work, this is a very rough draft but it works fine with the k-samplers.

[Pdonor]

#3192 here it is! I've been using this branch for the past few hours to make awesome pictures, this is a huge leap above what inpainting could do before! Even better, you can use it to spot-fix your txt2img generations before subjecting the world to them.

[f-rank]

#3192 here it is! I've been using this branch for the past few hours to make awesome pictures, this is a huge leap above what inpainting could do before! Even better, you can use it to spot-fix your txt2img generations before subjecting the world to them.

what's the "hands" situation like...

[chekaaa]

After doing a checkout to this branch , how do I use the new inpaint model, do I just download it and drop in the same folder with the other stable diffusion checkpoints ?

[bbecausereasonss]

Would love to know how to even checkout this branch.

[ghost]

git fetch
git checkout {branch-name}

[chekaaa]

no need to switch branch now , since it was merged to master 17 mins before this post

[bbecausereasonss]

RunwayAi just released 1.5 + 1.5 inpainting CKPT's.

[B34STW4RS]

Downloaded it and tested,

Loading weights [a9263745] from Y:\ASD\stable-diffusion-webui\models\Stable-diffusion\v1-5-pruned.ckpt
Global Step: 840000

seems to be working, gonna run some comparisons tonight and see if its actually legit or not.

[rycossette]

I'm a complete noob to this but was able to get the main WebUI working. But I'm honestly confused as to how to incorporate this different inpainting method. Is there a detailed step by step method to set it up, for idiots like myself?

[Ehplodor]

Please look into https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#inpainting-model

[kingwingfly]

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#inpainting-model

And checkpoint can be short as .ckpt, use it as you using your other models.