Possible improvements to DPM++ 2M Karras sampling

[hallatore]

I'm playing around with the sample_dpmpp_2m function in the k-diffusion repo.

I got a quality increase on my images by doing this trick/bug fix?.

I need help to test out if this is just a false positive that seems to work on my machine, or if it works in general.
Please test it out!

How to try out

1. Open stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py

2. Add the following code to the bottom

@torch.no_grad()
def sample_dpmpp_2m_test(model, x, sigmas, extra_args=None, callback=None, disable=None):
    """DPM-Solver++(2M)."""
    extra_args = {} if extra_args is None else extra_args
    s_in = x.new_ones([x.shape[0]])
    sigma_fn = lambda t: t.neg().exp()
    t_fn = lambda sigma: sigma.log().neg()
    old_denoised = None

    for i in trange(len(sigmas) - 1, disable=disable):
        denoised = model(x, sigmas[i] * s_in, **extra_args)
        if callback is not None:
            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
        t, t_next = t_fn(sigmas[i]), t_fn(sigmas[i + 1])
        h = t_next - t
        
        t_min = min(sigma_fn(t_next), sigma_fn(t))
        t_max = max(sigma_fn(t_next), sigma_fn(t))

        if old_denoised is None or sigmas[i + 1] == 0:
            x = (t_min / t_max) * x - (-h).expm1() * denoised
        else:
            h_last = t - t_fn(sigmas[i - 1])

            h_min = min(h_last, h)
            h_max = max(h_last, h)
            r = h_max / h_min

            h_d = (h_max + h_min) / 2
            denoised_d = (1 + 1 / (2 * r)) * denoised - (1 / (2 * r)) * old_denoised
            x = (t_min / t_max) * x - (-h_d).expm1() * denoised_d

        old_denoised = denoised
    return x

3. Open stable-diffusion-webui\modules\sd_samplers_kdiffusion.py
4. Add the following code next to DPM++ 2M and DPM++ 2M Karras

samplers_k_diffusion = [
    ...
    ('DPM++ 2M Test', 'sample_dpmpp_2m_test', ['k_dpmpp_2m'], {}),
    ('DPM++ 2M Karras Test', 'sample_dpmpp_2m_test', ['k_dpmpp_2m_ka'], {'scheduler': 'karras'}),

[biship]

Nice change, however the modified version is way to washed out for my liking.
Is there a compromise between the two extremes?

[hallatore]

You can adjust the 0.15 factor to anything down to 0.0. Try 0.10 or 0.12 maybe?

[kjerk]

I wouldn't mess with the generation process to control what is effectively just the simplest contrast change. Seems to be around 20% and you're adjusted. Something like:

# As an automated process (needs a flag)
from PIL import Image, ImageEnhance
# ... image created earlier
contrasted_img = ImageEnhance.Contrast(orig_image).enhance(1.20)
orig_image.save('{image_path_here}/###-original.png')
contrasted_img.save('{image_path_here}/###-cfix.png')

To automate it, or standalone:

# Dumb simple standalone script, change '/path/to/image.png' and run
import pathlib
from PIL import Image, ImageEnhance

if __name__ == '__main__':
    input_path = pathlib.Path('/path/to/image.png')
    output_path = input_path.parent / f'{input_path.stem}_cfix.png'
    image_orig = Image.open(str(input_path))
    contrasted_img = ImageEnhance.Contrast(image_orig).enhance(1.20)
    contrasted_img.save(str(output_path))

[Panchovix]

@kjerk I'm sorry, but where did you edit those lines/code, to change the contrast?

[catboxanon]

Those are standalone examples.

[bbecausereasonss]

Wow massive quality improvement!

[kjerk]

I was trying to get a better look at this myself to evaluate and did a blowup swapping comparison. Maybe not the best naming but B is for Before Change and A is After Change, using the OP's images, and applying 20% contrast to the after image as per #8457 (reply in thread)

B Before, A After (click for big)

[CoperNick8]

Looks like resolution change - I'm not saying that it it different resolution, but it looks like.

[Miraihi]

I've actually had that problem with seemingly low resolution. So that was the sampler's fault, who would've thought. I thought it's a Vae problem.

[danielos123ssa]

May I ask in what file did you apply the contrast fix? and how did you manage the paths

[CoperNick8]

I like result but I don't like 0.15 in the code. It could be some preference. I have idea to produce series of the images from 0.05, 0.10 to for example 0.50. to get better idea what is this about, but I don't have a time recently, so if somebody likes this idea, then I would be very happy to see the results.

[catboxanon]

Might be something worth bringing up in the k-diffusion repo? The old_denoised variable is not currently exposed, so making an extension for this would require a fork of the repo or a re-definition of the function. I believe an additional callback would also need to be created in the webui in the callback_state function, cfg_denoised_callback happens too late and only contains x.

[EfourC]

@hallatore
How did you get the generated before and after images so precisely matched?

When I did some quick trials, there were always more 'macro' changes between the images that make it hard to compare, rather than only fuzziness/contrast differences like in your examples. By macro, I mean things like foliage pattern in the background, slight shift in stance or hair pattern, and other minor features.

Edit: My test was with xformers disabled.

I didn't see as obvious a difference in the sharpness and contrast like in yours, but the macro changes I ended up with made it hard to really tell.

I'm wondering if the model/vae I used, or some other settings affects this.

Can you give specifics of a prompt, model, etc. so we can compare baselines? (I'll try some more when I have time this evening regardless.)

[hallatore]

Take a look at #8457 (comment)

[Centurion-Rome]

I made some quick tests. It seems to have also some impact on other samplers

Improved quality (wow! It feels like boosting resolution of 1.5 models to resolution of 2.1 :)):

• Euler a
• DPM++ SDE
• DPM++ 2M Karras
• DPM++ SDE Karras

Decreased quality(?):

• DPM++ 2M => big contrast loss after increasing contras image has better quality

Has someone else also tested this change with other samplers?

[Panchovix]

Which lines or how did you edit the code to apply these changes to DPM++ SDE and Euler a?

[Centurion-Rome]

not others as proposed by author ;)

[hallatore]

The sampler is used in some permutations. (The Sampling presets in SD are a mix of sampling + other stuff).
I can't remember exactly which ones use this sampler.

PS: Check comment bellow, I might have found something. 🤔

[Adreitz]

@Centurion-Rome Can you post some examples? Only @Panchvzluck has posted 3rd-party examples so far and I'm not seeing extra detail in them.

[hallatore]

I think I might have found a/the bug.
Please try this code and see if things improve.
I left the changed line commented out.

Replace this in stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py

 outdated ...

[hallatore]

@OEvgeny I tried DPM++ 2M but see perfect results here. Do you have info on how to re-generate that image?

[hallatore]

Here are what I'm seeing.

PS: Need to look at it with 100% zoom.

[hallatore]

Updated the post with proper instructions on how to add this as separate v2 samplers so it's easier to test.

[OEvgeny]

@hallatore I will try to follow the updated method to see if I can come up with the same result I posted earlier

[hallatore]

I see I get washed out results when "Always discard next-to-last sigma" is not checked in settings and using DPM++ 2M.

[Centurion-Rome]

We need to get this fix into official repo :)

[myndxero]

Lol, getting anything into the repo lately has been painfully slow. Dunno if the gitpython PR has been merged yet and that's been a critical issue for weeks! Guess I'll go look now that I'm here.

I respect Auto for the work he's done, but time to pass the torch or at least share it, if gonna be so inactive.

[mcmonkey4eva]

The change described is a shortening of the CFG Scale Scheduler option already included in the DynThresh extension - https://github.com/mcmonkeyprojects/sd-dynamic-thresholding

(EDIT: At the time this was written, this was correct - the opening post just tacked in a scheduler. However the opening post has since been edited to use an entirely different technique)

[Centurion-Rome]

Hmm I don't think it causes such a quality/resolution boost as the solution proposed here.
We need to test it also. Has anyone done this and may share the results?

[myndxero]

And I absolutely adore that extension. I'm still making test grids. Ultimately I think I'm going to leave the scale schedulers on constant. I've started to see a correlation between the ups and ancestral samplers and the downs and non-ancestral samplers, or maybe I have that backwards. Anyway, chosen to just zone in on a happy default spot for the thresh percentile and the CFG/CFG mimic relationship that I like.

Good to see you somewhere other than discord!

[hallatore]

Did a test with low steps. The difference is clearer at lower steps. I feel my version matches the 50 steps version better at lower steps.
PS: Click on image and view it 100% zoomed.

[hallatore]

Here is a version that works with DPM++ 2M. At least I seem to get pretty good results with it.

And with "Always discard next-to-last sigma" turned OFF

At 10 steps: https://imgsli.com/MTYxMjc5

 outdated ...

[fractal-fumbler]

can such improvement applied to another samplers? :)

[Panchovix]

+1 to that, I would like to see it also on DPM++ SDE, DPM++ 2S a and maybe Euler a, but maybe I'm asking too much.

Nice work OP!

[hallatore]

This fix/tweak only works for DPM++ 2M/DPM++ 2M Karras since it's related to how it uses the denoised part from the previous step. It's still work in progress as I'm not sure if I fixed it the right way yet.

I can try to explain what I think is a bug, and what I'm trying to do.

The code uses r = h_last / h which (I guess) presumes that h_last is always larger than h. At lower steps this can be false, especially with DPM++ 2M. So I changed it to r = max(h_last, h) / min(h_last, h).

The other thing is that it user r to make denoised_d a value between denoised and old_denoised. This is then used like this: (-h).expm1() * denoised_d
I changed this so it uses h_d which is a value between h and h_last. Kinda the same way denoised_d was derived.

I'm not sure the logic is correct atm since it's mostly been trial and error. So while I think I need an h_d value, I'm not sure I'm calculating it correctly atm.

Hope that made some sense :)

[fractal-fumbler]

thanks!

[hallatore]

Test with a human face

[catboxanon]

I think @mcmonkey4eva is probably correct. This new enhancement washes out the image when using it with img2img, which the vanilla sampler does not do.

[hallatore]

Did you use the latest code at the initial post? Mine is spot on.

[catboxanon]

You're right, I didn't know the code had been updated since then. Thanks for the pointer!

[FrisanBR]

I didn't find the mentioned paths, has something changed? I started using Stable Diffusion today, so I installed the current version.
"Open stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py"
"Open stable-diffusion-webui\modules\sd_samplers_kdiffusion.py"
I found the sampling.py file in a different path and was able to make the change, but I couldn't find the sd_samplers_kdiffusion.py file

Can anyone help me?

[axsddlr]

in honesty, if you cant find the paths, i wouldnt suggest trying this out until its considered for merge

[OEvgeny]

Is it the same repo? I thought EasyDiffusion is a completely different app not related to stable-diffusion-webui. And these instructions are for the ui available in the repo.

[2blackbar]

This should be in webui by default, so i dont have to put it back in everytime i update

[wywywywy]

We need to either fork the k-diffusion repo and become new maintainers, or we take the main bits from the k-diffusion repo and maintain them inside of this repo.

But we may break MacOS compatibility as they use a different k-diffusion.

EDIT: On the other hand, if we take control, we could easily incorporate the MacOS fixes into the new code, which will make it even cleaner.

[wywywywy]

Ok I've gone and forked the k-diffusion repo, and incorporated @hallatore's change plus the MacOS MPS workaround from @brkirch.

To use it, simply delete the repositories/k-diffusion directory, and set these variables in the batch/shell file:

webui-user.bat

set K_DIFFUSION_REPO=https://github.com/wywywywy/k-diffusion.git
set K_DIFFUSION_COMMIT_HASH=ca06f522e6d3f202c25c3565c53afbd9c40ac53d

webui-user.sh

export K_DIFFUSION_REPO="https://github.com/wywywywy/k-diffusion.git"
export K_DIFFUSION_COMMIT_HASH="e3f853a8c9f70052aa1c4bb8cd0e4ec3af7ffaff"

Please can someone with a Mac give a good test.

[nemilya]

If I replace whole repo by your repository (wywywywy/k-diffusion, ca06f5)

• Euler a not working
• PDM++ 2M working (better at small steps)

if use current (brkirch/k-diffusion, 51c977), and overwrite just sampling.py

• Euler a working
• PDM++ 2M working (better at small steps)

[wywywywy]

Any ideas what it could be? The sample_euler_ancestral function in the 2 repos are the same.

https://github.com/brkirch/k-diffusion/blob/c41588ae74e1b51e9f8663106d9843389bf02729/k_diffusion/sampling.py#L139

https://github.com/wywywywy/k-diffusion/blob/ca06f522e6d3f202c25c3565c53afbd9c40ac53d/k_diffusion/sampling.py#L139

[wywywywy]

Actually I just realised that I've missed out @Birch-san's MPS fixes. I'll get them added in and see if they work in this repo.

[wywywywy]

Ok new commit e3f853a8c9f70052aa1c4bb8cd0e4ec3af7ffaff seems to work. Please give it another try.

I was able to use both Euler A and DPM++ 2M Karras on my MBP.

[nemilya]

Just tested new commit - all works great now! Euler A and DPM++ 2M
Thanks!

[generatorman]

What does this change do at a conceptual level?

[wywywywy]

What does this change do at a conceptual level?

Basically @hallatore identified a potential bug in the DPM++ 2M logic, as explained here #8457 (reply in thread)

And from testing it seems to improve the quality quite significantly at low steps.

[hallatore]

The change does two things primarily.

• It makes sure r is always min / max. Since with some steps the next sigma can be higher th
  an the last even though they are normally always lower. This is what causes the artifacts seen especially in DPM++ 2M at low steps.
• The second thing it does it changing h to h_d in the last part. (This part ... (-h_d).expm1() * denoised_d). The reason for this is that denoised_d is calculated based on previous and current step, while h was just "current step". It made sense that both parts are calculated the same way, not only denoised_d.

[nemilya]

The fix works!

Machine: MacbookPro M2 (8Gb, MacOS, Monterey 12.6.3)
Sampler: DPM++ 2M, Steps: 10

Before/after:

I change file sampling.py directly in repository (variant by setting ENV did not change the repository k-diffusion)

ps: another confirmation, same 10 steps (before/after fix)

(examples based on Deliberate model)

[wywywywy]

Cool results thank you. However, colours seem to be less vibrant. Maybe more work is needed.

[thanayut1750]

can we add more contrast to it now? kind of like the result but like you said the colors seem the fade.

[bryanray]

it is not less vibrant, but LESS DETAILS, more blur..... it is not favorable unless you ask for it.

[myndxero]

If ever considered for merge. Yes I'm salty auto mia. Checking out a new fork that's supposed to be pretty active, seems to be getting a lot of attention. https://www.reddit.com/r/StableDiffusion/comments/12grgwh/automatic1111_getting_rusty_future_of_this_repo_i/

…

On Thu, Mar 16, 2023, 03:28 Andre Saddler ***@***.***> wrote: in honesty, if you cant find the paths, i wouldnt suggest trying this out until its considered for merge — Reply to this email directly, view it on GitHub <#8457 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2755JZHZOK34ASIXX7IS23W4LFLDANCNFSM6AAAAAAVVGRDVQ> . You are receiving this because you commented.Message ID: <AUTOMATIC1111/stable-diffusion-webui/repo-discussions/8457/comments/5330846 @github.com>

[LiteSoul]

I've been using this improvement for over a month... it's great, we need this officially somehow!

[recoilme]

@Metachs: "It looks to me like all you are seeing is faster convergence due to loss of detail" : crowsonkb/k-diffusion#56 (comment)
Take a look at this

[ride5k]

just stumbled across this, and am very happy to have found it!

[bryanray]

the result mostly blurring out lots details.... please fix it ? or is it unfixable....

[ShivaeAI]

I think there are similarities between this new sampling (DPM++ 2M Karras) and kohya_ss's Network Alpha training parameters in terms of adjusting the noise into blur and drastically reducing the noise, but using the flower photos sent from nemilya's comment above, I feel there is a blurred out beauty, yes, only a blurred beauty, no sharpening involved

[Alfonsusnetflix]

i seen that the script is updated ? any possible update ?

[Stalinium-Network]

where do you get the code for DPM 2M Karass, I'm new to this and don't understand how to find the code

[dtlnor]

you may insert the code here modules\sd_samplers_kdiffusion.py
or directly modify from kdiffusion (repositories\k-diffusion\k_diffusion\sampling.py)

note that the implementation here may lost details (crowsonkb/k-diffusion#56 (comment))

[simonmcnair]

Did any submit a PR for this ? Seems quite trivial to do ?

[mr-september]

It's 2025, I found this trying to troubleshoot why all my images seem to turn out blurry. Someone mentioned it's an inherent flaw of DPM samplers, and to go back to euler/heun. However, across pretty much all guides/discussions, DPM++ 2M Karras is still recommended for great balance of speed and quality, without any mention of this error.

It's absolutely mind-blowing that such a critical thing has remained unfixed across all forks/versions of the web ui for 2 years, and probably ComfyUI too.

Reading this thread feels like someone discovered cold fusion and a bunch of guys reproducing it and celebrating, then just forgetting about it.

[Birch-san]

I've never heard DPM++(2M) described as blurry. multistep samplers usually create overly sharp images.
Karras schedules especially should not be blurry; they spend many timesteps on high-frequency details.
native schedule could be blurry or at least erroneous because it spends no timesteps on high-frequency details, and will incur a lot of error in multistep samplers due to weird step size ratios.
if you are using "Always discard next-to-last sigma" then that would explain why your images are blurry, especially if you are on native schedule. you would be taking a large step during the few high-frequency steps in your schedule, resulting in a poor estimate.

absolutely mind-blowing that such a critical thing has remained unfixed
Reading this thread feels like someone discovered cold fusion

steady on.

the technique proposed here is based on questionable assumptions. the improvement it is seeing is because it is changing the sampler to be more like Euler. there is also a mistake in the implementation which is due to more questionable assumptions.

the core of the technique is this:

h_min = min(h_last, h)
h_max = max(h_last, h)
r = h_max / h_min

it's trying to always keep the step-size ratio greater than 1.
so the step-size ratio no longer functions as a step-size ratio.
no reason is given for why it would be bad for the step-size ratio to be less than 1.

to my understanding, no rationale was given in the DPM++ paper for how this coefficient was chosen (I didn't check the older papers though). I expect it was chosen empirically. it's an attempt to vary the weightings of the information from the current versus previous step, to better adapt the Adams-Bashforth solver to work on non-uniform step sizes (exhibited especially by the "native" schedule).

if the previous step was bigger (in log-sigma space) than the current step, r will be greater than 1. the larger the difference, the larger the r. when r is large, DPM++2M reverts to Euler behaviour.

so the reason this is semi-effective is due to "any location where the schedule is not linear in log-sigma space" (i.e. step size ratio is not 1), this technique will make r bigger and thus we get more of a Euler step than a multistep.

I am not sure whether the original DPM++ formula intends the behaviour (for r<1) that "when we take a big step after a small step: inform our multistep greatly with a second derivative from the small step". maybe it's fine, because small steps are more likely to be accurate. but I suspect the absolute step size is likely to be important too, not just the step size ratio.

but yes, if you disagree that "when r<1 we should really lean into the multistep" then a reasonable solution is "use more Euler-like behaviour". that said, I don't think "invert r" is a principled solution; you should just switch to a Euler step if you think your multistep would give you a poor estimate.

I don't know the motivation for how r was designed, so don't know to what extent the "lean into the multistep when r<1" behaviour was intended. @LuChengTHU suspects that during the high-frequency timesteps, the diffusion ODE suffers from non-Lipschitz continuity, which may be the real problem here (i.e. you should avoid multisteps during high-frequency timesteps). if this is the case, then what we're calling a "problem with r" might only actually be a problem during the fine-detail phase of denoising.

anyway, this problem doesn't exist for exponential schedules (r is always 1; the schedule is entirely linear in log-sigma space). and the problem is very small for Karras schedules (because they're very similar to exponential schedules).

this thing seems poorly-motivated:

h_d = (h_max + h_min) / 2

because it means that when we get the model's denoising prediction for timestep t, we don't actually step down to the noise level t. we overshoot or undershoot it. I would call this a bug.

anyway, the real solution here is "use Euler when step size ratios are extreme". DPM++2M's r variable basically does this for you for "small step after a big step", but you may want a similar behaviour for "big step after a small step" (r<1). the technique proposed here inverts r, which sort of does the trick, but it's probably better to just make r infinite or very large. or go down the Euler codepath that's already there.

the reason "Always discard next-to-last sigma" helps on native schedules is either due to extreme step size ratios or due to non-Lipschitzness for t near 0. but it is not a good solution; it's better to switch to Euler steps than to make steps larger. and next-to-last sigma is an arbitrary/dangerous cutoff point; better to express it as a sigma cutoff.

of course, all of this is immaterial because high CFG and high stochasticity ruin the estimates of multistep samplers. if you're not doing few-step sampling, just use Euler Ancestral.

[recoilme]

any manipulation may make image worst or better or more blurry as in this "fix", and in some cases - it will look better (by someone opinion - surprise).
You may multiply matrix on random number - and image will be different, and sometimes "better" - and open issue - and so on

Actually its not fix - it's "fix" - close this stupid issue pls