New method for realigning training may improve controlnet output too [method+code and proof of concept comparing AddDifference available] #1717
SwiftIllusion
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I don't know how the realignment is done here, the only reference I found was an article referencing the previous method that was a script implementing a simple add difference with SDv1.5 as a reference, that said it was outdated by this extension.
However, in the super merger extension a new 'trainDifference' method that makes moving the training of a model possible has been added, which could be applied to controlNet too.
For example this is revAnimated "man smiling" with AddDifference Anythingv3-SDv1.4
This is the same with trainDifference Anythingv3-SDv1.4 instead.
The relevant code from here
I was hoping it may be possible to implement this new method instead of just the standard add-difference as the above example shows a clear impact, and could result in improved outputs using controlNet.
I tried to integrate the idea idea into original tool_transfer_control.py to prove the difference with the controlnetv1 models, but couldn't work out how to get the script itself running (errors I couldn't resolve/not sure if I was running it correctly), and extract_controlnet_diff.py looks like it works differently.
I've also tried multiple times, and again now since first posting this to try and search through the controlNet scripts to work out where and how its interacting, but couldn't find anything that looked like the tool_transfer_control script or any standard method via searching for 'key' etc. Closest I got was finding "aligned_adding" because removing changing "base + x" to "base" gave the output as if there was no controlNet, but I couldn't work out how to manipulate x/total_controlnet_embedding worked, let alone understand how or where I could apply the above, or if it's possible without the original models like in controlnet v1.
Beta Was this translation helpful? Give feedback.
All reactions